Most data work seems fundamentally worthless | Hacker News Reader

Most data work seems fundamentally worthless

ludic.mataroa.blog

605 pointsludicity310 comments

Working in the ML/data space and have been fortunate to have largely steered clear of this problem so far. My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment.

Jobs like this exist, though you may have to take a pay cut compared to the bullshit. My first job after finishing my PhD was as a staff scientist in an academic lab using ML engineering + data science to support scientific research. There seems to be a fair number of grant supported jobs like this and pay isn’t terrible. Just under or just at 6 figures. You can make more in industry, but scientific work feels very meaningful.

Now I’m working in credit risk modeling, something I never expected to be doing, but so far it’s been a good fit. The models are applied directly in decision making for the business, and there’s a real incentive to get everything right because mistakes could harm real people’s lives. The team I’m on is strong and ethically sound and I feel good about what I’m doing.

For anyone in the data space who’s despairing about the state of the industry, non-bullshit jobs do exist, you just have to look for them, and use your judgment when scoping out new roles.

I really like your heuristic, and I'll add to your list of examples where one could work

- Any tech company where the Stats/ML model is one of THE products and differentiators. You'll need to cut through a lot of buzz word and sales-speak to find the good ones.

- Banks and other financial institutions where making uneducated guess is a big no-no when it comes risk, pricing and anything related to financial products. Your example of Credit Risk Modeling is a classic example and very interesting problem.

- As much as consultancy gets a bad rep due to some shady practice from big players, there exists a solid demand for professionals that know Statistics in the Large Construction Projects space. Let's say modeling demand and return financial for a project, proving environmental impact, preparing/implementing/analyzing unbiased surveys in the area, and so on.

- Government agencies where data is one of the Key Outputs. Such as the Census, Bureau of Labor Statistics, CDC, and so many others.

As you said, sometimes it will mean a pay cut, especially if you want to remain in the technical work and not deal with the business and managerial side of things. But there's solid demand.

+1 for government. I work with state and local governments a fair bit, and while there’s a lot of red tape and weird politics, the overwhelming majority of clients I work with are smart, capable, highly mission driven people doing their best in a system designed to move slowly. Being able to look back on a project and see a positive impact in a community is way more rewarding than trying to move the needle on click through. And the pay cut for public sector consulting isn’t as much as one would think. Especially for federal.

I’ve been doing “big data” in the retail space for almost a decade now and it’s pretty clear to me how it affects the bottom line. It really can be as simple as “we need more of this milk” and “the spicy queso is very popular”

I'm at least somewhat skeptical. The face of the last data warehousing fad in the 90s was around things like optimizing retail through things like putting diapers and beer [1] close together. But pretty much, even when there was some correlation, stores never did much about it.

[1] https://canworksmart.com/diapers-beer-retail-predictive-anal...

I can imagine that one reason is that you also need customers to be able to find things, which is difficult when not sorted into some meaningful categories. That said, I think at the Target near me the beer and the diapers are relatively close.

Actually you want customers to not be able to find things, so they have to wander around, and 'find' other things (in large displays for example) that you want them to buy. If the beer and diapers are always together then they won't discover the new high-margin sausage rolls. So what's better for the customer from a correlated data perspective may not be better for the business from a maximizing profit perspective.

> you want customers to not be able to find things, so they have to wander around, and 'find' other thing

This seems to be the Loblaw/NoFrills model in Toronto. They relocate products with alarming frequency.

Want milk? That's in the dairy section. Want organic milk? That's in the health food section at the opposite end of the aircraft hangar sized store. Nuts? Either in the snack foods, health foods, or baking ingredients depending on unknown critera - one literally has to check all the places.

Taken to the extreme though, I'm going to be taking up employee time asking where something is and eventually stop shopping at a store if it's too frustrating.

To be clear, the linked article debunks the whole thing. The correlationa were largely apocryphal, spurious, or useless.

That's not quite what the article says. Although it doesn't explicitly say so, there was apparently a (possibly valid) correlation. But Osco didn't act on it, instead removing a bunch of slow-moving SKUs instead.

Though that also makes the point that making things easier for customers to find is probably more important than any minor co-purchase optimizations.

I think there have been more recent fads. (If fad is the word for it.) There was the story about Walmart stocking up on Pop-Tarts during hurricane season that was in the news fifteen or twenty years ago.

There was the original "Big Data" and Hadoop, Chris Anderson's "The End of Theory" and Clive Humby coining "data is the new oil" around the same time. Probably others.

This is funny to me. In the old days, this was called: cost accounting to determine the profit of any particular item you sell and market analysis to see how many of which things your customers are likely to buy. Join the two results to optimize your profits. It's not rocket (/data) science.

Actually neither of these is particularly easy to do for unorganized companies.

I'll really would like that the reatailer prepared for me my montlhy basket based on my previous and recurrent consumption.

My household would really like it if Amazon foods merely provide the same list we bought form last time to adjust for this delivery — we absolutely hate having to start from scratch every time (and that's not even mentioning their issues with substitution).

Heck, the local wine & beer outlet lets me just open up the last purchase list and buy that again (and adjust the quantities if desired).

WTF is wrong when a local store can get it so right, yet Amazon, with it's emphasis on "always be hungry like a startup", so totally forks it up?

Substitution was the real issue I had with online grocery delivery when I last used it 15 years ago. Things would get substituted that I didn't really think were equivalent and essential ingredients for some meal would be left off the order.

It was "OK" at the time; I was on crutches and could go to the store but not easily do a full grocery shopping. But I haven't done online grocery since.

15 years is a long time in tech

It's a great way to make sure people never try new things.

How so? It could include new recommendws things.

Most online grocers do this.

And if stores had meals pre-shopped in bags you could just remove unwanted items from. Grocery stores seem ripe for simple optimizations even without big data

Grocery stores would be better served by not being giant chains with optimization problems, at least in North America.

Better zoning laws that allow mixed-use zoning would enable more, smaller grocers embedded in neighbourhoods. Personalized service would be much easier on a smaller scale. And you wouldn't have to travel far to pick it up.

Aside from specialty shops, I'm not sure there's any great virtue in smaller markets, often with higher prices and less selection, other than there are probably more of them you can walk to if you live in a denser urban area. And my observation at least in the UK is that most of those smaller grocers are giant chains like Tesco and Sainsbury.

Being the giant chain is the optimization. Grocery stores have razor thin margins; operating at scale is the only way to keep prices reasonable.

This is also one of the few domains where people are very price sensitive, and they regularly see and think about costs. I couldn't tell you what my local automotive guy charges for an oil change, but I can absolutely tell you the price of flour and cheese.

No one is going to pay $10 for a single apple, and if you raise prices hard enough people are going to start gardening.

This is a recipe for significantly higher grocery prices which is the exact opposite of what you want for one of the staples of stability: food.

"... this is just the rise of this new general store, and how all specialty stores look the same now, so none of them are actually really that special."

https://www.marketplace.org/2023/02/02/why-are-boutiques-and...

Most people don't want to pay extra for personally-provided grocery service.

If the giant chain (or a network of smaller grocers) had good personalization, it would make the stocking the small local pickup point more practical.

I live in a diverse multicultural community. A small local grocer can't stock everything my neighbors and I want.

It's funny that we are always chasing after 'personalization', which is of course only a problem created by industrialization and massive scale.

I believe you, because the store is always out of the most popular products, and nobody ever seems to figure out that they could simply shift production towards the more popular products and make more money?

You sure about that?

On a local basis I can go to two of the same store a few miles apart and one will out of products X, Y, and Z, but the other store may be out of X, T, and R. Localized buying trends can have significant differences.

Also, there may be many other effects here. For example, if a popular product is actually popular and the pipeline to make the product is months long, well there's going to be outages. Shifting production generally isn't easy and trends pass quickly.

In addition, perceived popularity can be used to manipulate consumer pricing. "X is always out. Oh look X is in stock now, I should buy it for 50% more"

Yes, I am sure. I go to grocery stores in multiple locations and they are all much more likely to be out of a particular version of a product than other versions. It has been this way for years. They do not price this version differently than the other versions. And this is true for multiple products even in different kinds of stores.

I think some suppliers just aren't that good at adapting supply to demand.

> for years

prior to covid? last few years have not been representative about how well the supply chain works over the past 50 years.

> My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment.

Currently starting to look for a new role and couldn’t have articulated my goal more clearly. The problem is that this narrows the field considerably, to finance/insurance/fraud (“pure money work”), healthcare/EMR (which as far as I can tell is also actually just profit optimization work for hospitals), and then you have manufacturing (closer to where I currently am), but the actual applications of data science are more limited and data Eng / analytics are more valuable, bioinformatics (which is really interesting, but also a large learning curve), and then just traditional BI (kind of generic and boring).

Also, if you’re thinking about long term career development, many of these functions fall under the IT/CIO umbrella as orgs grow, which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc. which I’m not saying is a bad thing, just an observation. Just applying statistics and modeling can only rise so high in an org, unless like you said, their core business and value prop is being the best at modeling some phenomena.

I’m finishing my bioinformatics-centric PhD and I really resonated with the author’s sentiments. Except in the case of the bioinformatics field, I feel like there’s too much data for any person to make sense out of. And if you do anything beyond a volcano plot, everybody’s eyes roll back into their head and they get bored and ask for their volcano plot or their special gene/protein/transcript of interest.

Tbh I dread my future job of maintaining R and python scripts

anti-fraud is very unsexy work IMO, but as long as you believe that (insert some internet-enabled service) should exist, then it can't be safe and enjoyable to use without serious fraud mitigation and prevention work, you can be secure in knowing that you're making life a lot easier for people who otherwise would have suffered from that fraud

Curious why you think antifraud is “unsexy”? I worked in the field for a few years. My experience was that the bulk of fraud is easily detected with bog standard models (most people doing fraud are lazy). But, there’s a long tail of super sophisticated work for detecting more subtle operators. It’s also pretty fun to learn about the convoluted schemes people come up with for doing fraud and to imagine how someone would attack a new service.

Also theres quite a bit of money sloshing around and it’s relatively easy to quantify financial benefit from fraud reduction, so the departments tend to be well funded

unsexy just in that it's not like a net new feature thing you can point a random person to like "I made that!"

Like security, it's a field where success is when (externally) no one knows anything ever happened.

> which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc.

I was going to suggest: we have a bunch of data scientists working on intrusion detection problems in cybersecurity, tied to an engineering organization instead of an IT organization. On the balance of considerations, I think it has a net positive impact. And I'm pretty sure plenty of other companies have something similar. Unfortunately, we also have an ongoing hiring freeze, so I can't recommend my own organization.

When I was starting out my career, my father gave me a piece of advice that has turned out to be incredibly applicable and valuable - "Stay close to the money". What he meant by that is work in environments where the work you perform is very closely tied to the revenue of the company - or another way of saying it - "Do work that is directly part of the organization's mission". It sounds like you've applied this to your career with good results.

Yup, always try to work in the profit centers, not the cost centers. Was reminded of that a job or two ago in a big way.

Every time I don't listen to Patrick McKenzie, I am filled with regret.

Yeah, I was basically filtering out jobs applications by whether or not I could see what effect ML was supposed to have on the functioning of the business, and importantly, whether or not they actually had access to the data to make it work.

Ended up at a place where ML doesn't just make things better, but is necessary to make things work at scale.

There were a lot of job ads I looked at that just seemed dreadful to me though. So many recruiting companies wanted ML Engineers/Data scientists. I could kinda see how it would work, but didn't think I would enjoy it.

As a non-ad tech example, a significant component to what we're trying to do at https://www.auxon.io is provide tech to companies they can use for testing & analysis of robots and cyber-physical systems. From our perspective internally what we're getting is the ability to perform something akin to "materials science" for the increasingly critical software parts of these systems and construct models of their behavior to use for predictive & comparative analysis purposes.

We have a research partnership with the University of Ottawa & University of Luxembourg specifically on the statistical analysis end of things to go deep in some areas to later incorporate the findings into our products. In fact the first go around of that research cycle has already happened (https://arxiv.org/abs/2301.13807v1) and the insights are being integrated into our Deviant product (https://auxon.io/products/deviant).

It's definitely not ad-tech. It definitely has a specific applied use case. Most of our marketplace traction is in aerospace, energy, automotive, and defense. We're not immediately hiring for roles on the data analysis end of things (we're in much more immediate need of visualization & frontend help), but we will be this year.

Focusing on jobs where your output makes an immediate impact is an incredibly smart move to make sure you matter. Of course, most roles matter but there are too few managers who know how to vouch and articulate a team’s business value properly.

I actually found myself working on a credit risk modelling project on the capital markets side and it’s been great as well.

This is actually the most important career advice for tech jobs.

First junior dev job it's not as important because you just need to find something. After that you will want to be very careful to make sure the job has actual business impact, otherwise you are likely to end up in some kind of bullshit vanity project that only has to appear to work.

If you're a baseball player, you don't want to work for a football team. You'll be undervalued and hate your job.

Less metaphorically, I've always looked for jobs where my skills are aligned with the "company mission" - first at a bunch of startups, and now as an academic researcher where I get to define that mission.

Well said! To add a few options

- energy management (shifting loads to times when energy is cheap) for consumer/commercial/industrial use cases

- energy markets, especially power trading: often highly algorithmic, and driven by models that turn fundamentals data (weather, calendar, …) into supply/demand predictions, and from there into price predictions

- retail pricing, both offline and e-commerce

Can you tell us a little about the models and software packages of credit risk?

Is it trying to take a huge dataset of consumer features and join it to a dataset of loan outcomes and then predict loan outcomes?

Yes, that's a common approach. You can take a dataset of consumer features at the point in time when loans were opened along with information about the loan outcomes and try to predict the loan outcomes. You can't just take a kitchen sink approach with the features though because there are regulations that demand a level of explainability. To get a sense of the basics, I think the book Intelligent Credit Scoring by Naeem Siddiqi [0] is very good.

[0] https://towardsdatascience.com/book-review-intelligent-credi...

This is a really good answer

i would also like to add that modelling in credit risk is not just about yes / no answers around loan outcomes.

There are lots of other goals that are regularly modelled such as default rates, profit optimization, loss minimization, delinquency and payoff rates at specific parameters ... endless options

There are also lot of different ways in which these models are implemented ( decision trees, statistical analysis, ML... )

Some examples of (real life) projects include:

- if our institution offers this card to clients with 750 credit scores vs 790 credit scores, how does my profit move vs my losses and what the factors to limit losses while maximizing profits

- how do I minimize my costs for servicing this card while keeping profits at the max ?

- what rewards options lead to the highest number of preselected / qualifying clients taking up a product at the lowest cost

- what contact strategies are best for specific types of clients if they are late on payments - call or email or text or legal letter? which strategies are the cheapest? which strategies give what this institution considers to be the best response ? which lead to fastest full payment? fastest partial payment? which lead to getting back to a regular payment plan?

- how can we identify clients who have a lending product with us who might be on the market for another lending product in the next 12 months? in the 6 months?, those who might need a limit increase pro-actively? those who whose might need a limit decrease pro-actively?

And, one of the largest area pf credit risk evaluation is real time decisioning on transactions: 'is throwaway201606 really buying $6000 of apple products, in person, at this mall in Toronto, Canada right now when I (the system) know I he bought a daily Wendy's Spicy chicken sandwich 10 minutes ago in Dallas" and should we allow this payment

Some example of how models are used here include ( note that modelling helps establish which transactions to look at more carefully and which to ban outright among other things )

- predict what type of terminals are being targeted: scammers -> we have left bank ATM machines alone and started looked at gas pumps:

- predict where transactions of interest might come from: scammers -> we do scams on site A at Christmas, scams on site B in the summer or we do site A scams with brand Y card and do site B scams with brand Z card

- predict behaviour patterns of transactions of interest: we always test the cards we will use by purchasing a $5 'brand x' gift card online 10 minutes before

Yes, completely agree, I omitted the part about not just modeling binary outcomes for the sake of brevity. I was even considering linking the article Capital One: Exploiting an Information-Based Strategy [0], showing that one of Capital One's innovations was successfully modeling more complicated outcomes like profitability.

[0] https://www.computer.org/csdl/proceedings-article/hicss/1998...

I agree they do exist, and this heuristic sounds sensible to me. It's the good old patio11 "go work in a profit center" situation, and I wish I had done so. I'd probably be grappling with a new existential concern, but that's life (I wish I had a job that didn't matter so I wouldn't have to worry about performance all day!).

> a clear and immediate impact within the organization

This reminded me of those Microsoft Viva emails I keep getting. Anyone finding those useful for anything at all?

To me it seems like a solution in search of a problem.

That definitely goes to the category of products that exist to keep a highly paid data/development team busy

I find it strange that AI/ML people would avoid "adtech for ethical reasons".

I really wish the websites I visited made better use of my actual history on those sites to tailor relevant ads to my interests. It seems like an ideal application of AI to me. They could do a far better job than serving me the lowest common denominator stuff they keep throwing at me.

My Twitter news feed these days:

* 10% - posts of interest from people I follow, i.e. the stuff I actually go to Twitter for

* 90% - Ads and recommendations of topics to follow that I have zero interest in

If it's going to insist on showing ads and recommending topics of interest, you'd think those could be better personalized, given that Twitter has years of my tweet, reply, and like history to train its AI on.

But no... what I get is crypto ads, Hollywood events, celebrity news, sports news, etc.

The problem is that there are a loooot of ethical implication on using your own personal data in the first place, where that goes, who has access to it, how is it handled, and so on and so on. Then advertisements isn't anything but propaganda, which has its own set of implications. And then finally we have the ever present pressure to push more and more ads, thereby making the internet in general worse and worse, so the very field of ads is in itself unethical, as it is destroying the virtual environments we are building.

Also ads != recommendations. In a sense after a while these two are also at odds with each other. Cause there is again, the ever present need to sell you more stuff.

>I find it strange that AI/ML people would avoid "adtech for ethical reasons".

You founded and run an advertising company. Are you taking the piss? Surely you've aware of the ethical issues even if you don't give a fuck about them.

Or are you just saying you categorically expect AI/ML people's interest in the tech and/or money to override the ethics?

Are we doing ad hominem attacks now?

Perhaps we can start from the points I'm making and assess them on their own merits instead?

>Perhaps we can start from the points I'm making and assess them on their own merits instead?

As far as I can tell, the points you made are:

A) It's strange people make moral choices to stay out of advertising.

B) Advertisement could be more targeted.

B is boring and I don't care.

A isn't really a point with merits to be debated.

I enjoy eating meat. But I don't call it "strange" everytime someone turns out to be vegetarian. I understand why they refrain from eating meat. It's not like I disavow the existence of the question of "should we eat meat?". I just disagree.

> ad hominem

Be careful before you start tossing around the Latin, someone might hit you with a 'cui bono?'.

I find it strange that anyone would want companies to accurately model their behaviour and desires so that they might influence you to behave in a way that benefits that company.

The rare time an ad slips past my blocker, (and rarer still when I stop to think about them) I take solace whenever they are irrelevant.

> I find it strange that anyone would want companies to accurately model their behaviour and desires so that they might influence you to behave in a way that benefits that company.

If the companies in question are billing the ads by impression, as is common, they get paid whether the ad served is relevant or not. If I'm going to be served up native content or ads anyway, I far prefer they be relevant to my interests than they're not.

> I far prefer they be relevant to my interests than they're not.

These companies are trying to extract my attention and money -- both limited resources -- and if I have to see ads better ones that I have no interest in, simply out of spite.

I'm failing to see how this addresses the concern of corporate entities trying to effectively shape my behaviour, with no regard for my well being, to favour their interests.

I'm not sure what particular interests you have in mind, but if I visit a site I don't have to pay for, subsidized by advertising, and that shows me content and ads relevant and tailored to my interests, I consider my own interests pretty well served.

Is it possible for us as people to model our own behaviour so we can influence ourselves in a positive way?

Tracking body weight, wearing an exercise band, recording personal bests, counting calories, keeping a journal, counting number of days sober...

Depends on who defines "positive"?

> I find it strange that AI/ML people would avoid "adtech for ethical reasons".

I've been doing ML for over 20 years and if it were up to me, advertising would be illegal.

Sounds pretty extreme. Telling anyone about a product you've built is really a form of advertising. I don't see how society advances if your viewpoint were to be implemented and taken to its rational conclusion.

It's almost impossible to be competitive whilst remaining ethical since being unethical provides a massive advantage. In the drive to optimize profits, any hard advantage soon becomes table stakes.

Not to argue for OP, but I think it’s more of a slippery slope thing. A friend of mine does data science at an online bet site but doing more of the number crunching; and from what I gather, the advertising side get close to pandering to gambling addicts

Edit:ignore, op posted. I should stop jumping to answer so soon!

You understand that you're not the one paying for the service, so you don't get a say, right? Even once The Machine has perfect knowledge of you, the ads will not be tailored to your preferences. It will be what an advertiser has paid The Machine to show you.

> what an advertiser has paid The Machine to show you.

As Cory Doctorow has point out so eloquently in his "enshittification" series, the end point isn't even to the benefit of the advertiser. At the end state, The Machine also have perfect knowledge of the advertiser, and the adtech company can turn the full power of the The Machine to the benefit of itself, extracting value from both the audience and the advertisers.

Actually I do get a say... take the Twitter example. In that case it's resulted in me visiting Twitter far less than I used to.

I may not be voting with my dollars as a user here, but I am voting with my attention span. Dollars are not the only asset of value in play.

Please come to the field of computational biology, we need your help finding molecules that cure cancer and other diseases

Long-time non-profit guy here. Totally get where the author is coming from and I'm in a very similar position in terms of where my role relates to my organisation, but I still love my job despite having exactly the same issues. Why?

Non-profits are a funny old space. In theory, they're not out there to make money, so people gravitate towards them seeking "purpose". Everything is fine once you realise that you're not going to get that, or at least in the way people come to it for.

Non-profits have ridiculous, completely unachievable goals, particularly when you put their budgets in context. Some provide truly excellent services and products (e.g., reports), most don't.

So what's so great about a non-profit tech job? I've got an interesting problem space, I love trying to work out how to measure things that are inherently difficult to measure. I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't.

I bust my arse because I'm interested in what I'm doing, but I could easily coast if I wanted to. The pay is enough for me, and my colleagues are for the most part super nice and interesting. If I want to learn something, I can make an excuse to play with it at work and run with it.

Seeking to have "impact" through a data job at an average non-profit is naive, but a lot of these jobs have stuff to recommend them beyond that.

----

Edit: slight addition to my list of joys of non-profit nerding

I've been in non-profit now for a few years and this rings true. We are a non-profit "business" that competes directly with for-profit companies. We try to impress upon the public that we are non-commercial, when the reality is we are just marginally less commercial because we still need revenue to operate and do normal revenue operations to get it. We just won't gate or restrict our offering no matter what and mostly avoid any commercial interference in the sanctity of our mission. And yeah, we're trying to rebuild our data systems for like the third time in 5 years and can barely even eke out a plan as to what to measure let alone what to action on. On the plus side, no one is deluded and everyone wants to do better.

> I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't

As someone who's now the only tech person at a small non-profit, this really really resonated with me (as did the rest of your post).

I feel it's often challenging to asses how robust/brittle something should be implemented because we don't sell (and maintain) software. A lot of "tech things" simply have a very short life span, often closely tied to project scope and duration.

I hope you don't mind me asking here, but it would be great to connect with someone in a similar position. If you're up for it, please feel free to shoot me an email, you can find the address in my profile :)

Thank you for the insight on this. It's true, I know that some non-profits that are good. Over Christmas, I had the privilege of spending time with an incredibly smart person that that set up a successful non-profit helping out healthcare workers during the pandemic. It really struck me how this person still turned up at the office day-after-day to help out, doing manual labour if necessary, even though it clearly wasn't necessary - they could bask in their early fame and the opportunities it had brought if they wanted to.

But that definitely did not seem to be the most common case, as you've said. I think maybe I might write some quick edits to make this clearer, both to myself and reader, but 'most data work' being clueless is the real issue, not 'all data work'. That is, I've moved around jobs quite aggressively, but if there's a 10% chance that any job will be okay for a while, that could take quite a few moves to find one, and you can't change over that often!

> you're not going to get that

If you believe what the organization does furthers your purpose, then you are going to get that, although I agree that the day-to-day feeling may not be that "purposeful".

> Non-profits have ridiculous, completely unachievable goals

There a gazillion kinds of non-profits; some do, some don't.

If your non-profit is the town museum management foundation, you have a perfectly achievable goal (it's right there in the name).

If your non-profit is "the coalition to end world hunger", then maybe not so much. Then again, even such a non-profit might have the actual, stated, legally-filed goal of delivering food aid to areas in the world hit with natural disasters like floods or drought - and that's not an achievable goal in the sense of being ever done with it, but it is achievable in the sense of being able to do just that continuously.

> So what's so great about a non-profit tech job?

That there is almost never the situation of a huge pile of closed-source code developed internally which you have to live with; and no custom expensive hardware. So typically you need to both rely on, and contribute to, free software that basically anybody can use.

> Seeking to have "impact" through a data job at an average non-profit is naive

On the contrary. There is great potential of technological impact outside the organization due to what I said above; and there is potential for impact on what the organization does, because it's often more fluid, and using data, you can make convincing arguments about steering activity in a different way - appealing to your colleagues desire to serve noble purposes.

For what it's worth, I've mainly worked for "the coalition to end world hunger".

I've got a particularly fun gig at the moment, but generally tuned out of the mission a while ago.

I guess my point was that sure, as the original post suggested, this sort of work _can_ be meaningless when viewed as the author does, but there can still be lots to love in the sector.

> The absolutely fucked up thing is that everyone I've met in this space seems to have totally given up on doing anything meaningful at work. The goal is to get paid, not stress out, have a happy office where everyone can collect their strange handout, and not think too deeply about how unfulfilling is it to produce nothing for forty hours a week.

this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.

UBI would just socialize the cost which at least now, the private sector partly pays.

UBI is a yearly recurring expense in the trillions.

Lots of people would work less, (remember COVID?) lowering tax revenue.

This means higher demand for workers, who will require even higher pay because of supply constraints and high taxes.

Unlocked demand, decreased labor supply and higher taxes means inflation. Lots of inflation, specially in services.

Businesses will offshore or automate to try and reduce cost, but that will push even more workers onto UBI.

What will the new equilibrium be? Nobody knows. It could stabilize with extreme income inequality creating a permanent underclass, It could drive the United States into hyperinflation, creating civil unrest and destabilizing world security. Nobody knows. It would be a society wide experiment that would be politically impossible to undo democratically.

If you think BS jobs are bad, how about no jobs in a bankrupt United States, sliding into Venezuelan hyper inflation?

Imagine China, Russia, and North Korea undeterred and the death and human suffering that would follow.

The US may not be perfect, but the world is a harsh harsh place and people have no idea how good they have it right here, right now.

So good that professionals not only demand high pay from their employers, but think it's totally reasonable to require "meaning" as well.

Were he still alive, I would pay good money to see the author try and read his essay out loud to Victor Frankel.

I just experienced something like a flashback, I was in debate club, listening to yet another kid claim that social change X would lead inevitably to global nuclear annihilation. Strongest case possible, right?

We can't possibly re-allocate the trillions of dollars the federal government spends, or the more trillions the collective states spend, because it would lead inevitably to mass die-offs comparable to soviet-era famine conditions.

Existing UBI programs don't directionally move that way? Pandemic productivity actually rose, despite a couple of quarters of setbacks? The actual evidence doesn't support gloom and doom? Just make the claims of potential harm bigger! Nobody can counter hyperinflation, civil unrest, or world war!

>Pandemic productivity actually rose, despite a couple of quarters of setbacks?

Covid did not create Universal Basic Income. Everyone knew that Covid money will run out, so most people didn't quit their jobs (although still many did, which was GP's point). Also, the amount of debt raised during covid was no joke, and we don't know the long term effects. But even disregarding that last point Covid isn't a useful datapoint for disproving GP's claim. The fundamental issue is with UBI is people will not want to work and that is GP's central claim.

So GP using involuntary unemployment numbers in support of an anti-UBI case didn't trigger your alarm, but me using voluntary employment numbers to demonstrate the falsity of the original claim did?

If Covid isn't a useful datapoint for disproving GP's hysterical claims, then it definitely isn't a useful datapoint for supporting them.

The number of people who quit their jobs to depend on "Covid money" rounds down to zero. There just wasn't enough "Covid money" for that. Many people were unemployed during the early days of the pandemic, but not voluntarily.

Your "fundamental issue" lacks evidence and the counter-evidence from limited trials of UBI points the other way. But carry on. Civil unrest and a world war await.

Why does everybody who earns above ~$30K/yr go to work? They could retire after a few years if UBI would suffice.

Well, Chicago started a trial program of it recently. Doubt that would have come out if it weren't for the pandemic.

Debate clubs always lead to global thermonuclear war.

That's a very slippery slope you got there!

A slightly different version of the arguments you write here could have been made against pensions or a 40h work week when they were introduced. I'm not saying UBI would not massively change society and potentially include risks, I am just questioning your certainty that implementing it would lead to disaster.

Oh, and it's Viktor Frankl by the way.

this is actually not supported by any data. its pure speculation. plus the handful of UBI trials don't show the catastrophic breakdown of labor supply you claim to inevitably happen... the reality is until a proper ubi trial is done with actually livable wage we don't know what would happen...

Costs don't disappear just because it's the private sector that has them. On the national economic level, waste is waste.

But I agree that with UBI people who have actually important jobs might also quit, or demand higher salaries.

Some would say that sounds fair and sensible compared to the current status quo.

About software development specifically: Note how many of the most valuable software projects have grown not from profit driven corporations, but from open source maintainers working in their spare time.

A lot of your reasoning requires a big [cite needed] - specifically the "lots of people would work less". UBI pilot programs have proved the opposite. Also several states cut unemployment benefits to goose their active worker numbers and the opposite happened - less people working.

I'm not saying that nationwide UBI would be a good thing but your logic isn't clear.

> It could stabilize with extreme income inequality creating a permanent underclass

That should would be a frightening and radical departure from today.

Ok, but that just means the UBI will be very small. Nothing you said goes against the concept.

Lol, wow.

>The US may not be perfect, but the world is a harsh harsh place and people have no idea how good they have it right here, right now.

Amen.

> people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices

Private money being wasted due to inefficiencies at the office (as you say, a misallocation) is surely entirely different than a government-granted right to not produce anything yet still be supported.

I think this discussion rather gets to the bottom of the objection to UBI: nobody objects quite so strenuously to wasted work as they do to "idleness". In the view of these people, it is morally superior to pay people to dig holes and fill them up, or the white collar equivalent of digging holes in spreadsheets and filling them up again, than it is to just give them money with no strings attached.

The important thing is that people are prevented from enjoying themselves for at least eight hours a day.

No, not at all. The important thing is that the person making poor choices bears enough of the cost of those choices.

Otherwise, they will not grow.

True UBI would create the largest, most indolent, and most socioeconomically isolated consumption class the world has ever seen — until it collapses under its own weight.

Sorry to just jump in here, but growth isn't for everyone. If some people don't want to work, I'd rather pay them to get out of the way than risk the chance of them adding negative net value.

The people "making good choices" here being the unproductive office workers?

From a society point of view, it doesn't really matter "who" wastes it. Ultimately, the outcome is the same: Someone spends time on something that society does not benefit from.

... sidenote; I personally think that something completely inefficient "is not that bad". Much worse is work that is directly detrimental for society as a whole, even if it does make a few people rich. Now that is terrible.

Someone does something that society does not benefit from and earns a paycheck.

They then buy food, housing, medical, transportation, fuel, entertainment, (optionally) raise kids, by things kids want, take vacations, have hobbies

They're still most likely a net positive to the economy.

businesses fail, government programs are forever.

> Private money being wasted due to inefficiencies at the office is surely entirely different than a government-granted right to not produce anything yet still be supported.

Why?

If you are in the camp that think companies sole purpose is shareholder profit then this is perverse.

If you are not in that camp you can probably agree that we should aspire to the best life for most people.

(As a European this comment really dissonates. I do realise that it does not for Americans)

Private money wasted usually leads to that entity dwindling away from competition, with the end result (of efficiency) that benefits society at large. Public money wasted takes a much longer time to dwindle away and the end result is usually catastrophic for society at large, i.e. a revolution.

As a European, odds are that you're used to a monoethnic country where "most people", "society", and "government" are kind of interchangeable, and people find it easier to sacrifice themselves for the greater good -- as you say, aspire to the best life for most people, even if it might mean that the decisions being made are not optimal for one's own circumstances.

I myself hail from such a country (though outside of Europe), and you are right that the American viewpoint, or that of any other large, multiethnic country, is quite different. Implementing policies that require taxpayers to support others that are culturally different and therefore harder to empathize with is an order of magnitude more difficult; and it is becoming yet more difficult in America as time goes by, due to the reduced emphasis in assimilating to the mainstream ("white") culture.

Ultimately, these systemic inefficiencies are financially supported by people, the public at large.

At different rates though. You can absolutely choose not to pay the lazy-tax and buy from a different vendor that doesn't employ an army of people who do nothing, you can even build whatever it is you need, but you can't easily choose to not pay the taxes that the government collects for the same purposes.

Pretty sure Atlassian has a lower per-employee efficiency than what a smaller startup providing a product in the same market would have, but there might be specific sticky features in the Atlassian product that make it difficult for one to migrate. The effective market hypothesis is great at finding local maxima.

Likely, yes, but it's "do I want to work around this", not "there's no way". There's enough people getting fed up with Atlassian that "Like Atlassian X, but sane" could be a category.

There's basically no way you can start a new state if you'd want to opt out of the existing ones, and even moving from one to another is quite the undertaking. Imagine Atlassian saying "sure, you're free to stop working with and paying us, but we believe that we've been instrumental in building your company, and we believe that entitles us to 20% of your company if you want to work with someone else".

Even if that were true, and I don't think it is, there are millions of jobs that are subsidized with public money and the public has no meaningful choice in them. Think defense contractors, public sector consultants, etc.

"Jobs paid for by public money" and "completely unproductive jobs paid for by public money" are not the same though.

But even if it actually was millions of completely unproductive jobs, I fail to see how that would be an argument for adding a few dozen millions more and not a reason to get rid of the ones that exist and cost us money that we could otherwise invest into important things that are currently underfunded.

Can you really? Are you sure? I sincerely doubt it.

we already have that.

go buy treasury bonds, they pay almost 5 percent. ibonds pay more. why? they just do. and companies and rich people buy them all the time.

People don't object to UBI because they don't like the idea of people getting money for no work.

They dislike UBI because either

a) It's financially impossible. Assuming the basic income is "enough to live a basic life", say £1500/month in the UK, that is totally impossible.

b) They don't like the idea of people getting free money and not having to work when they do work. I mean you see that already with benefits/welfare, but UBI makes it more extreme.

Although obviously with UBI the people paying lots of taxes have the option of stopping work and moving somewhere cheap. Which is what a huge number of people would do, and the whole system would collapse.

But yeah, nobody is against a magical utopia where nobody has to work. They just know it can't exist.

Everyone I know who works in these kind of jobs are financially secure but extremely depressed.

One of the mistaken beliefs underlying the UBI argument is that the fulfillment part will come automatically for most people. Who wouldn’t want to spend every day painting, writing, or playing music? The problem is that most people who are not ideologically self-motivated are unable to find sources of fulfillment without some sort of economic influence.

Are the workers depressed because they are "not ideologically self-motivated", or because they must spend time doing something they don't see the point in doing?

It’s not because they didn’t see the point in what they were doing. These people had a lot of free time and only worked a couple of hours per week.

The late David Grabber had a great book about this - Bullshit Jobs. And when you draw the line, we do keep a lot of work and a lot of jobs around purely because if we don't a lot of people are going to have a lot of free time to ask a lot of questions about the way that the world works. And the very foundation of our system is that you have to go work, sell your labour to make someone else rich, then buy things to make someone else rich and eventually die.

> this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.

Absolutely. I've worked in multiple different companies, of different sizes, since about 2011 with breaks in between. One common theme was the amount of pointless work going on, stuff that could've been streamlined or automated, or manual reports that no-one reads or acts upon.

I agree with you, but it's not like you can wave a wand and trade UBI for bullshit jobs. The bullshit will persist.

If you’re a data professional in a company where data quality is low or non-existent, then improving the quality of data in the organization is absolutely part of your job, most likely the most important one

It’s less cool and flashy than the real time online machine learning model you want to build, or the multi-level bayesian model to determine causality between revenue and an obscure event, but that data dictionary and table/db catalog really needs to be built before everything else

Then document the essential reports to the extent that a high schooler should be able to produce/maintain them

Then adjust the reports so that the metrics indicate what management actually needs to learn from them (if you don’t know what’s important for them and for the business, then finding out is part of the job, too)

All these things are extremely important, and it’s silly to suggest your work is meaningless just because it’s not fun or interesting. There are no tv series about civil engineers building sewage networks but public sanitation saves more lives than medicine. Data work works the same

> then improving the quality of data in the organization is absolutely part of your job, most likely the most important one

You're absolutely right, but it can be hard to get management to bat for this kind of stuff. The biggest hurdle I've seen with getting "good" data is fixing the issues involves other teams prioritizing the work on their backlogs. Depending on the company and where the teams lie on the org chart, fixing things might require a multi-quarter directive from the CTO. Which isn't happening.

so then ... confidence intervals are going to be so use as to be useless. and every second week the data science team can present a 2 slide presentation, where the first slide says bad data = bad result, and the second says good data = good results, and then furiously flip between the two during the Q&A.

and everybody clapped!

This is already what I and the others I've spoken to do (who haven't given up). But, at the types of organisation I'm describing, it's near impossible.

You know that Airflow instance I mentioned? Three years later, that team still doesn't have anything to schedule Python workloads, despite extremely smart people trying to angle the politics of the damn thing to finally get traction for months on end. When you've got no leverage, you've got no leverage, and you don't want to spend years of your precious life arguing so that at some point, in a few years, you can do your job properly.

I absolutely agree with what you're saying, but I suppose I'm saying that, at many jobs, you're better off leaving and somehow locating the places where you have a chance at accomplishing what you've laid out. There's a sliding scale of incompetence, and some organisations are close enough to competence that you can actually move them the right way with some diligence and effort.

That isn't possible most of the time working for a larger company. Too often the problem with bad data starts not at the collection stage but the data is bunk to begin with.

Since the data is not under your control -- it belongs to the "real" engineering team, which has vastly greater status -- you will not be able to. You will complain and they will ignore you. You will offer to help and they will refuse. You will open a PR and they will reject it.

I’m dealing with the opposite experience right now. I work at a big tech company and lately we’ve been working closely with a data science team. I’m desperate for some guidance on what sort of data pipeline and practices we can set up for them and have a hard time getting back anything but “can you export some rows as a csv”. Which is actually easier said than done considering data compliance policies and what not. Maybe there’s a good reason but it gets a bit frustrating and I can’t imagine it’s not slowing them down as well.

You know it's a badly run team when they keep pulling you off your self-built, self-motivated ETL packages to breathe down your neck about incremental, barely-there releases.

Thanks for the space to vent a sec...

Too bad data scientists rarely have the skillset necessary to even begin doing that work.

The author seems to extrapolate from their experience of having a “six figure data job straight out of college clocking out at 5pm everyday producing no value” (paraphrasing) that all “data work” must be producing no societal value.

I don’t like throwing around out the phrase “privileged/out of touch” too often but this post doesn’t seem fit for the top position of HN.

I disagree, the author has touched on something widespread and hard to articulate that I suspect a large part of the HN readers have had experience with.

The problem is that lots of organisations say they want to be "data driven" but then leave all the management decisions to the prejudices of individual managers as well as management as a whole within the organisation. It's a problem that's been with us since Taylorism and the dawn of "scientific management".

You end up with "we spent years and $ to get the data which says do X, but we don't feel like doing X, so we're just going to ignore it because data in and of itself has no power within the organization".

It's like buying a gym membership and not going to the gym. Having a data science department satisfies the organisation's need to believe it's self-improving.

I found it resonated (although agree the title is click-baity, they just talk about their own experiences in multiple orgs).

It is hard to do good data work. Offhand it takes some combination of:

- business understanding and goals (that don't themselves come directly from data)

- using data to effectively orient those goals to the best opportunities

- using data to measure whether you are successful or not

Part of that can involve meta goals -- our data is not sufficient now to meet bullet 2, so we need to start doing something different to measure it.

I find many people in these roles act like they are just human machines producing reports. You really need to take agency in many situations IMO, and direct higher ups to look at data in the right way. If you just wait to be told what report to produce, it will not go well.

Being a data scientist is what I imagine being a lawyer is for idealists who go into the profession. They think there is an underlying reality that holds bad actors to account—for attorneys, this is the institution of “The Law”—but, in fact, most of the job is helping those bad actors justify what they already wanted to do anyway.

And just as most legal disputes end in settlements, most data scientists are excess capacity, kept around because the programmers who will put up with typical dev nonsense aren’t smart enough to hit the high notes… when the fact is that said high notes only need to be hit very, very rarely in business. Being a corporate lawyer comes down to intimidation—no one wants to face off against Apple’s team of lawyers—and 99% of being a corporate data scientist is talking in maths to impress (or defraud) clients and investors.

Care to elaborate where the value created for millions of people is in an industry that pivoted years ago to focus on ad revenue and psychological tricks to extract money from 'whales'?

Sure, advertising is collectively a multi-trillion dollar industry propelled largely by computer/data science, but it is still only a fraction of overall economic activity driven by advancements in data analysis.

The past couple decades have seen huge advancements in safety, reduced workplace injuries, auto injuries, etc. all driven in some part by analytics.

This sentiment that “all data analysis = advertising = evil” seems very reductive. It reminds me of all the comments I see from my technical peers about how “useless” other departments are such as HR and Sales, when they’re fundamental to a healthy business which pays the salaries of programmers.

Most of jobs in academia are like that. Everyone knows it, some few admit it, most do not want to rock the boat.

Having worked in data in many different environments including market research, Fortune 100, unicorn startups I beg to differ.

It's extremely accurate, though. He described 4 of the last 6 jobs I've had, as if he had been there with me.

The CTO of my previous company was fond of saying something like:

> According to [some book he read once], the average company stands to increase their profits by [50%?] by applying data science. Even if [author] is only 10% correct, that's still tens of millions of dollars for us.

(some variation on this at every all hands meeting)

When I left the company, they were still desperately searching for that first 1% of benefit after nearly a decade of effort. Like most management fads, there is some solid foundational truth there, but it quickly becomes a solution in search of a problem when it gets tossed at every single problem faced.

The trick is to discover what the current management fad at your work is, and then reword whatever problem you ACTUALLY want to solve as being a case of implementing the fad.

I've known of very successful middle managers who had multi-decade projects that were really advancing the actual work of the business but the top-levels thought it was various separate buzzword projects during those years.

That's a pretty poor understanding of basic prob & stats. What if you are not an average company? What if you're not an average company and on the low end of that bell curve? If the author is only 10% correct, maybe the author is only 10% right about the type of company that could benefit from "applying data science".

Very early in my programming career I still couldn't decide whether I want to do programming or graphic design. I worked for an online game written in ActionScript (Flash) and was designing armors and some other in-game objects. I also participated in the game's forum on behalf of game's admins where they collected and discussed player's requests and suggestions for in-game new features.

My ICQ number was thus available to the wide audience as I was sometimes the focal point of player's complaints or requests. One day a player I didn't know personally contacted me on ICQ, and his avatar featured a piece of headgear I designed. I had tears standing in my eyes knowing that someone valued my work so much they actually decided to use it to represent their online identity.

Never since have I been so close to the situation that whatever I did mattered. My wife works for the government (also in IT sector), but she has better job satisfaction than I have in the sense of knowing that whatever she does has any impact at all.

In my case, I'm automating testing and deployment of a piece of software that's essentially exclusive in its market niche. So much so that customers have no idea how bad it actually is, since they have nothing to compare it to. The testing of the software I work on is outright worthless, so whether it's more automated or not -- it will have no impact on the outcome. Not to mention that the biggest problems with the software are in its design, in the decisions that went towards the core elements of it, that, at this point, nobody dears even to question, let alone to reverse them. So, it sucks. It's going to suck. And it's going to suck for a very long time in the very same way because no replacement is coming.

But, hey, I get a visa! And a permanent contract! Yee-haw!

Sometimes being able to make something suck just a bit less can be powerful, and in ways you may never see.

Very true! The times when I've been happy at work have frequently been when I've failed at basically everything in my job description for the week, but managed to write up a script that automates some horrific process that a colleague I like was forced to do.

Would the organisation actually keep me hired if they knew that was my main output? Absolutely not. Did it make me feel great? For sure! Taking note of those small moments has been helpful in working out exactly what I (and many others) find distasteful about office work.

Some of the "most productive" times I've had were when I was sent to a customer site to diagnose some esoteric issue, and got to watch them actually using our software/hardware, realizing there were some unreported issues that made their life harder but nobody realized, and quietly fixing those (alongside the main issue, which was a configuration edge case) for the next release. Never even hit JIRA, but someone's life was made easier.

I've lost that hope / illusion a while ago. Today, I spend days away in pointless attempts at convincing someone to fix a bug that I know full well is not going to be fixed, just so that I could clear my consciousness (or cover my rear) by pretending I've done my job.

I'm 53 and have been in the IT Security space since before there WAS an IT security space. I have installed 6 SIEMs, just put a bullet in SIEM number 5 (oddly the first time I'd ben in an organization long enough to both install and uninstall one.) and the Current generation has an on-prem and off-prem installation due to data sensitivity...we're re-doing the on-prem installation for reasons...so I'm kinda at SIEM 6.5

of the first 5...NONE of them are still in existance. I recognize the circle of life nature of my job and have gradually changed my raison de etre from 'Protect the company!' to 'Advance my career!' to 'Pay off my mortgage.'

It was a bitter pill when I realized my job was 'look at magnetic patterns on a spinning disk on a computer half a world away and determine if they were GOOD patterns (company data) or BAD Patterns (bad guys)'.

I'm going to take an educated guess that you've just turned 30 or are about to turn 30?

I had this exact same problem where I became entirely disillusioned with the output of my work after having become reasonably senior and realising that the majority of my time and effort ammounted to projects which either failed or treaded water. The unfortunate reality in business is that the projects that truly change the world are 1 in 100. They don't look or feel much different from any other at the begining but when traction really does start to take off you know you have something right. I think from your description, you need to work at a startup where you have genuine impact. This is the only way you truly "feel" meaning. You might still build something that's reasonably worthless to humanity, but you will do it your way and take your own risks.

I felt disillusioned 4-5 months in my first job after college. I could see the project I was working would take a long time to create something of value (3-5 years) but we were always put under pressure to show some stupid demos to management every quarter.

So at the end we were just doing these demos and not working on actual things that we needed to do. The project was important, if implemented successfully it would cut jobs of ~60 people in the organization. So management kept on adding more people until they realized that there was no real progress and they dumped the project and we got reassigned.

I realized that 1 year of my work as well as ~15 other engineers just went to waste.

I am surprised there are no private equity like entities that are driven by a single thesis which is to buy non-tech public corporations (or as far away from tech as possible: like commodities and real estate) then cancelling all 'digital transformation' or 'data engineering' contracts with consultancies like McKinsey, replicate the important 10% part of the contract in-house by recruiting a solid small tech team.

Then take the company public later at 10x the valuation, having slashed the cost by 10x without impacting operations or growth.

Knowing which 10% is the important bit. The problem with these projects is that you need to extract the tacit knowledge from the lower level staff in the company, who have often been incentivised for years to keep it to themselves.

"Manage better" is not a scalable system. Although it does work for some individual takeover merchants; it's a big part of the Warren Buffet success.

This is not supposed to be a scalable model, it depends on deep domain expertise of enterprise IT and also would require some hands-on active management and most importantly recruitment. So it is not your usual private equity play "slash everything by 30%" gain short-term profit, flip it, then watch the company close down in a few years time due to morale and lack of growth strategy.

A semi-generalisable strategy to identify the 90% quick wins in consultancy contracts: look at the consultancy contract, anywhere it says Oracle, IBM, Microsoft, Data Science think about how you can replace that with an open-source stack run by a competent technology team that you will recruit and what impact scrapping all the data science stuff will have.

The crucial bit in this strategy is to offer a package to this new elite tech team that would be competitive with FAANGs, this will be completely surprising to most management of these companies as they are used to paying their tech stuff garbage when in effect they are paying 10x FAANG salaries via the daily rate of McKinsey consultants.

It would be interesting to see it tried. It's effectively "digital transformation" with the host/mutator roles reversed. Rather than have an organisation call in outsiders to do the transformation, the transforming group "eats" (buys) the transformed organisation.

The risk is definitely in the squishy unknown unknowns that are hard to quantify and therefore get obliterated early on in a digital transition.

If you figure out how to identify those 10% and get the business on board with following that, I’d suggest skip the buying companies part and just sell your consulting hours at triple McKinsey rates.

McKinsey doesn't get brought in on the basis of their competence even though it's of course part of the image they project. If you want to go with quality work you'll have to buy your way through because the upper management doesn't have a clue about tech and you will not be able to justify your higher rates if you're a nobody.

Executives at those companies understand that risk and set up all kinds of landmines to scare private equity away.

A simple example would be, they could put the IT support department under the "VP of Crypto" instead of under the "Director of IT". So private equity comes in and fires the entire crypto department because they assume it is useless and then find out that no one can get their passwords reset or computer replaced.

More complex examples involve finance. You could transfer stock in your company to a banker as collateral for a loan with the condition that if the loan is paid off early there is a 1 billion dollar fee. Taking a company private requires that the buyer buy all the public stock.

Is it really that common to be bought by private equity to see it as an ever present threat?

Are you able to point to resources that document some of those examples? Just curious.

Anti-takeover measures have been around for 40+ years and generally are referred to as "poison pills".

https://en.wikipedia.org/wiki/Shareholder_rights_plan

https://www.investopedia.com/ask/answers/042315/how-can-comp...

I'm not about to praise the Musk takeover of Twitter as a good thing, but he did prove you can make massive cuts in a workforce and keep things running just fine. And he's emboldened the rest of tech to do the same.

It's not just data analysis. There's bloat all over tech.

> and keep things running just fine.

For at least a couple of months, yes. The jury is still out as to how his actions affect the long-term sustainability of the company.

If you were a farmer you could stop buying seeds entirely and save a hell of a lot of money. And you'd be perfectly able to continue to grow and reap the current crop. If anything, productivity would increase because time spent buying and storing seeds is freed up to focus on the current crop.

But next year...

It would take firing and replacing most of management...

And when you look at it, yes, there are plenty of funds driven by buying a company, replacing the management, and selling it. Most of them are not long-term driven (you really expect people that flip companies to be long-term driven?) and thus make all kinds of decisions that you'll probably disapprove.

I would qualify this as tech/IT management, but you will not need to touch the management team that is running the core business. In fact in most cases even a big chunk of the IT management is outsourced to third-parties.

Vista Equity Partners?

Unfortunately this post is basically the author just showing they're young and out of touch.

Here's the thing-- some data work is fundamentally worthless. Yes, some companies are a total mess when it comes to their data. Some roles are tasked with projects that are... to put it bluntly, a waste of time.

But it's your job to be improving the status quo rather than wringing your hands and declaring it all pointless. This is frankly where the difference lies between someone more junior and someone that can be an effective engineering leader-- they're looking to constantly improve.

Yeah, that means making a case to leadership sometimes. Being an effective engineer (software/data/ML) ultimately requires strong people skills.

There's an undercurrent of victim blaming here.

A lot of companies hire data scientists simply because it's "the thing to do" and they want to appear modern in the marketplace. But then these data scientists are typically left to work alone and put under pressure to "produce results" with no tangible understanding of their mission.

Sure, you could say that it's up to them to advance their own case, but if you have no support system (which is something that most people need) then it's very demoralizing. Unless part of your job is to figure out how and whether a data driven approach is even applicable, then this is can be a very depressing situation to be in.

> Unless part of your job is to figure out how and whether a data driven approach is even applicable, then this is can be a very depressing situation to be in.

You make a good point... what is your job? I suppose I've been coming at it from the perspective that it very much is a data scientist's job to figure that out. Sometimes the answer is, no, using ML for X use case is a waste of time. Or "no, there's a qualitative heuristic we can use that's better than some lengthy statistical process". At most serious orgs it's expected that "no, this is a waste of time" is a reasonable answer.

The issue of not having a support system is an orthogonal problem no? The reality is some companies and even teams within good companies don't offer that. So you have to learn how to navigate politics, etc. to get execs to buy into your vision/results/suggestions.

Author is NOT the victim. They have a lack of people skills that could be remedied by taking some proper coursework, or seeking professional help. The companies author is working for are the victims.

> Being an effective engineer (software/data/ML) ultimately requires strong people skills.

This is something I wish people had emphasized to me earlier in my career. It doesn't come to me naturally, but it's essential, and I wish I had started building these skills sooner.

Writing code (or analyzing data) is what sets your job apart from other jobs, but it's not where you provide the most value. You provide the most value at the intersection between your specialty and what the rest of the business is doing, and in order to provide value at that intersection you have to be able to understand what people need, adapt your work to those needs, and then influence them to adopt your (now actually helpful) work.

If you can't do that then your work is all happening in an ivory tower.

Yep. And this is ultimately, IMO, what separates a "programmer" from an "engineer". Same could be said in the DS space I suppose.

At a certain point in your career the easy part becomes the coding. Anyone can learn to code... it's really not that hard. It's understanding best practices, keeping up to date with the state of the art, knowing how to solve complex problems (not in the sense that the problem is necessarily novel but in the sense that you must do so pragmatically with existing data models, infrastructure in place, and legacy crap), and how to work with stakeholders both above and below you that pose the greatest challenges.

This is somewhat the reason I switched careers from data science into data engineering. My experience was that most companies weren't mature enough to really make use of any sort of "real" data science. But there was always some data sources sitting around somewhere that could be copied around, cleaned up, then thrown into Looker. After a while, this collection of data sources will turn into a data warehouse, and a network effect will kick in, making looker (or tableau/etc) broadly useful for other people in the company.

Most of the teams I work with need simple reports to do their job. There's no need for predictions or self-learning classifications. Mostly things like, is the information in these three systems consistent? If not, can an alert be sent to notify someone?

That being said, even my current company has 10x as many DSes as DEs and I'm pretty confident that all of them are like the author of this article. I can't blame them for taking a better paying, far more prestigious job with several times as many openings in the industry.

To be fair, as a DE I feel no different than the post author. I build pipes that might as well be going to /dev/null

> Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other. Sure, they might betray each other constantly, but they won't challenge the social fiction that some sort of meaningful work is happening.

One of the best summaries I’ve ever read about this phenomenon.

This resonates. I've moved from data engineering into sales engineering. The variety of problems is fun, albeit the context switching with Account Execs that don't respect my calendar is... something.

A lot of my previous data work was someone saying "show me what's interesting"... "no, not that- something else"... "huh, this doesn't jive with my gut, start over".

Bringing _value_ out of data sciene / engineering is incredibly hard. You have to have the engineering skills as baseline, but I firmly believe everyone undervalues the amount of work to "sell" your analysis to less technical folks in a way that's inline with the needs of the org. It's incredibly difficult.

Reminds me of "Goodbye, Data Science" from a few months ago:

https://news.ycombinator.com/item?id=33787270

The main reason I soured on data science is that the work felt like it didn’t matter, in multiple senses of the words “didn’t matter”:

https://ryxcommar.com/2022/11/27/goodbye-data-science/

For part of my career over 10 years ago, I worked in data science for big tech, and I can definitely see the hazards ...

It is extremely easy to get hoodwinked into joining a useless data science team, one that is mere signaling for executives, ineffective, ignored, or otherwise

OMG, pretty much a lot of what I have personally been through. But the problem is deeper and goes beyond a company not knowing what to do with someone that does ML/Data Science ... Most of software that comes out either does not contribute to society at all or is in some form harmful to it, usually due to the need for such things to be somehow profitable. And pretty much all the senior devs I know are burned out on this in one form or another and want to quit and have a farm/bakery or whatever that actually does something

I suspect you could remove the word "Data" from that title sentence and it will still ring true to many people.

Maybe replace "Data" with "Information Technology". Most people don't work in information technology, and occupations like teacher, construction, health care worker, lawyer etc. are certainly not worthless.

Startups are a good place to look for data jobs where you can make an impact. It's pretty easy to use data the "right way" if you start early, but it's incredibly hard to change the way data is used at 30+ year old companies or ones that are already hundreds of people large.

If you're the first data hire at a company of 10 to 20 (mostly engineers) and the founders seem sharp, there's a good chance you'll be doing things that actually help the business.

There's obviously a risk that the company will go under, but if the answer to "would I use this?" is "yes", that's been enough for me to be happy at work.

Worthless would be great. I struggle to find tech jobs, especially in data, that aren't worse than worthless - i.e. actively making the world worse.

Thanks for the chuckle.

> The institution produced a fair amount of academic research, but the vast majority of it was essentially fraudulent. We had some contracts with industry to produce technology in the space, but these mostly seemed to consist of the organisation fumbling the projects and massaging the truth (read: lying) until they could collect more funding.

Wow, this is very honest, which is refreshing in today's world. Thanks for the post OP.

I'm a developer (age 50+) rather than a data scientist and I've been lucky enough to enjoy all my jobs.

Partly I put this down to working at small companies where you have the autonomy to get things done without too much bureaucracy. However if I was put in a situation where I was working in a large organization and felt the job wasn't making the most of my talent, I would do 2 things:

(a) take my best guess of what I could do that was in my power that would have the greatest impact on the bottom line, and spend as much time as possible doing that (this might not have much relation to my job description)

(b) go as far up the management chain as necessary to try and get someone to understand what I was doing and why.

This maybe a naive and optimistic approach but I'd like to think that after you've tried this with a few different employers you'd find somewhere you were properly appreciated.

I joined a scale up in the BNPL space as head of data engineering.

It was one of the most poorly managed IT environments I'd ever seen.

There was so much technical debt, that basic systems where literally dead or dying (day 3, Tableau stopped working), and while I did my absolute best to address it, the constant overarching priorities were always more data, and faster data. Reliability, security, performance was my problem to deal with. (As well as literally anything technical, yes even configuration of mail clients).

The demands for solutions came thick and fast from the head of data science and where always clear as mud, yet I was pushed to the wall on immediate and unquestioning commitment for data projects and was held accountable for dates regardless of the resources, capacity, dependencies or technical debt we were fighting on daily basis.

Attempts to push back on any commitments or seeking clarification or coming back with a different solution were often met with my manager telling me, publicly in meetings, that I was going against company principles. At other times the head of data science would get extremely angry and storm out.

I ended up having to track our team's time, and I remember at one point in time in a quarterly planning session (that they had just transitioned to ) we could actually see how under resourced we where because based on our estimates (backed by data, tracked meticulously by time) of projects they expected in that quarter we could justify a 10x increase in our resources.

This apparently shocked them and us so much, they ( my manager and some other data representative) penned a pretty crazy letter to the CEO basically saying all the work the team had been doing for the last X years (since I joined) was wrong and bad and etc etc... completely unfounded. I only found out about it because the CTO raised some eyebrows on some of the claims. I ripped it to shreds pretty much questioned every claim with counter examples, and they had to throw it away.

At that point I quit.

A BNPL company demanding instant gratification and ignoring the cost of debt? How ironic

Watching growing startups hire a junior data scientist to analyse their new mountain of data is very entertaining. One of their first findings is their revenue/growth seems to be dropping… “we need someone to focus on growth/revenue ASAP”. They hatch a plan, build a team and magically their growth starts to recover.

The issue was spotted in April/May, the building happens during the summer and is launched In September… I’ve watch this cycle happen so many times, I’ve given up trying to point out that the issue is just seasonality.

I worked with a guy who loudly pointed this out in executive meetings where marketing and analytics were presenting. Then he wouldn't let it go, loudly proclaiming the sham in the hallways and through various emails.

He was laughing at them in their faces, even, and was 100% correct. They excluded him from any further meetings.

Every time they brought up a new stat and set off alarm bells he would bring up their massive fuckup and rub it in their face, and then would dig through the data and question every point.

He was eventually fired for pointing out how incompetent our new owners were after an acquisition, and would have been a lot happier joining everyone in their sham.

Yeah, I haven't seen anyone do this firsthand, but all my social intuition tells me that you can't challenge the social reality being presented. You can yell at one manager, call his team incompetent, so on (all deeply inadvisable), but in my heart, I don't think anything would get you strategically blackballed and removed faster than saying "So... what do we all produce, exactly? Oh? You realize that's absolutely worthless, right?"

I feel like the title is a bit clickbaity. This post is more of a rant against poor management than actual data work per se. That being said it's true that unlike other roles such as developers, artists, copywriters, etc. that still can produce tangible elements of the product under poor leadership, data will be the first victim of poor management and will have a higher probability of not producing anything at all.

Data buy-in within a company is absolutely essential for it to work.

I rarely comment on HN but this article struck a nerve as I have worked at a handful of toxic bureaucracies that suffered from this issue back in the day. It's incredibly demoralizing and yet every place like this eventually reaches a weird equilibrium that preserves this mess as this unspoken pact amongst the people who are getting paid for nothing (queue "Office Space" scene - "what would you say you dooooo here Bob?").

If you're not in debt and massively dependent on the job my best advice is run, don't walk, to find a place where you can get paid for value delivered. Startups are generally not afflicted by this cancer because there's simply no place to hide- everyone is doing six jobs and it would be nearly impossible to conceal this level of inefficiency. And if it ever did proliferate in a startup then that's _definitely_ not some place you want to be. It sounds like you've gotten a few bad dice rolls but rest assured there are plenty of good startups out there creating important products, paying their employees and valuing their work.

As challenging as startups are, I would way rather take that environment any day over the slow boil (or the slow bake following the lasagna analogy) and gradual erosion of my soul at the Office Space job.

One other thought for you: if your circumstances dictate that you _must_ stay in a toxic job environment like this and there's simply zero way to change the environment or leave, partition your work life into a box so that toxicity doesn't permeate the rest of your life. Clock-in, tolerate the nonsense, laugh about it with colleagues then clock-out and find your meaning from other pursuits, whether that's spending time with your family, taking up a hobby, doing volunteer work, whatever. It's great when your job produces a sense of meaning but if meaning is absent from that, you can definitely find or create it elsewhere. I've spent the past year building up https://Problemattic.app for just this scenario. I was fortunate to have finished a 26-yr career in various tech roles and now have the luxury to try and work on solving this issue. I'm convinced now more than ever that it's possible to find a profound sense of meaning by applying those professional skills that are currently squandered and redirecting at least a fraction of them towards the pursuit of solving important societal issues.

Anyways good luck. We can always use data science people so feel free to peruse the projects and contribute a few data science cycles to anything that grabs you. Or propose a project and rally a team if you have an idea for something not currently listed. cheers

As someone figuring life out (and also the OP), just wanted to say that I really appreciate you taking the time to write this. Advice is always great. For the most part, this is largely what I've been doing after a lot of analysis, though I avoided including my post-realization strategy in the article as... well, firstly, I haven't seen results yet, and secondly, I don't have enough experience to guide people through this. Some people are fine with the work as long as they can spend time with their kids, and I respect the hell out of that.

I've got a few engineers also looking to find a way out, and we'll definitely check out your platform!

In scientific research, data mining as it's apparently often practiced in the private sector isn't a thing - there's study design, there's controls, there's a whole system of how to go about collecting data that's been designed in advance so that datasets are collected that can be used to answer the questions of interest.

Garbage in, garbage out, that's worth remembering.

Here's an example from my wife: they are working in cooperation with some hospital. They were tasked with extracting breathing information from the EKGs. (This could be helpful in detecting when to take patients off the ventilator). Worked for half a year with some previously acquired data. The hospital part of the team aren't particularly tech-savvy and didn't participate in the research as much. But once they actually had to produce current EKG data, it turned out that the sampling frequency of their equipment is half of the minimum required in order to detect breathing.

Now the research is worth nothing.

Another similar story where I got tangentially involved: my wife had to reproduce a notebook of some guy who wrote a thesis on detecting something in EEG signals. After a lot of digging and research into that guy's work, I discovered that the format they used to acquire EEG signal in could have either 32 or 16 bits per "event", the event streams were not contiguous, but per sampling per electrode attached to the scalp. I.e. physically, they were written as event1_electorde1, event1_electrode2, ... event1_electrodeN, ... event2_electrode1, ... etc.

The original research author didn't understand this aspect, and the data he used was generated by different EEG machines in different format. He used some framework function that tried to guess how many bits per signal were used, and when it failed to guess, it'd print a warning. He simply silenced the warnings. And that's how he ran his ML models, and that's how he submitted his thesis. He has a graduate degree now and had moved to a different country. His work is worthless.

I’ve spent some time in the data science world and can emphasize. I think one reason it feels like this is that you’re relatively close to the power structures of the organization: the CEO says everything should be “data driven” and you are “data”. That makes everything very political.

EDIT: And to state the obvious: corporate politics is hugely inefficient and mostly worthless.

Incredibly insightful take. I need to think about this. It's absolutely the case that a C-suite Acronym person will have an idea, and suddenly the whole organisation is scrambling to pivot it.

The reason that I mentioned vision is important is that at one of the roles, we had an extremely competent Chief Data Officer. While I think all their management were basically lying to them actively, they nonetheless managed to plot a clear enough course that the organisation has improved substantially. Is all the work still worthless? Yes, absolutely. Do code review, CI/CD, and cloud enablement all exist? Also, yes, absolutely. If we do this for another 5-10 years, we might actually get something done.

The other places though? Absolute chaos around everything where the word data is mentioned.

Corporate politics may be "worthless" but they are the manifestation of human nature in a large organization. Thus, they can't be avoided entirely, other than not joining a corporation at all.

Most of data work is just middle management paperwork at the end of the day.

The incremental value of the nth dashboard, nth model, nth analysis is decayed very quickly. What's worse is the demand side for "data stuff" is largely driven by ad spend and companies trying to get alpha on beating ad spend. So you have all this venture money getting burned up by ads, then burned up by all the headcount and compute of "data teams" trying to beat ads.

Some people in the thread have pointed out that this is like UBI.

I think it's actually more socially reckless and far more sinister than UBI. Given that VC funds are largely pension funds, essentially the pooled money of the middle class is what is subsidizing $200k+ salary analytics engineers, data scientists, as well as the $200k+ "influencers" selling stuff to the data engineers and scientists and analytics engineers and whatever other invented titles VC comes up with to get 2% management fees off laundering this pension fund money and university endowments into Snowflake credits so that to make "insights" can be made for middle management.

I write about these things here and there.

https://medium.com/p/bfd4b4e33ac7

I also did a podcast recently where we covered how pension money goes to Sequoia and how that resulted in a Reverse ETL vendor sending me a panini press bribe and how panini press GTM tactics are unlikely to result in returns back to grandma's pension funds. You might be putting grandma on the street by participating in this whole charade.

https://www.youtube.com/watch?v=yInWs4NUs0E&t=233s

It's too much nonsense in the data world. It's enough to make you completely insane.

I have to ask, is it normal to join a startup that claims AI powers their main product, and then join the company only to realize they aren't using anything remotely resembling AI?

Like expecting CNNs and vision algorithms, and finding out they are using off the shelf OCR software? So their core product itself is bought off the shelf?

Yes. It is conventional wisdom to assume that all startups are lying though their teeth. Claims about what they've accomplished are understood to be what they hope to achieve eventually.

One of the first jobs I was offered was for a startup in SEA, which I didn't mention in the article. It was an AI powered trading platform, and they revealed in the interview that they were going to launch without AI and just do all the trades manually. A few friends have seen similar, but less well-funded, ventures do similar things. So yes, absolutely common.

not uncommon

"The more things change, the more they stay the same" - Jean-Baptiste Alphonse Karr

In the late 1980s, "executive dashboards" were all the rage. Vendors were selling tools to tap data in mainframe databases to provide insights directly to execs to improve business knowledge. It generally felt like a massive waste of resources.

(But, in that timeframe, I did build a tool to scrape data from a mainframe DB and turn it into useful statistics on a weekly basis at a big organization. Not sure how long it lasted, though, because my boss required that the reporting interface be implemented in his favorite tool, Lotus 1-2-3.) (edited for spelling)

You could say a lot of the work is speculative because you might be wrong about how much value it ends up producing. But if you say that 1 project in 10 pays off you just take the cost of the other 9 as the cost of the 10th.

That why this kind of department for sure was called R&D... His organization just didn't reached the "D"...

Worse than the ones that are fundamentally worthless, are the ones that seem worthwhile until you turn them off and find that nothing changed. Which is far more of them than you'd expect.

I’m supporting a $30B companies ERP migration to S4 as a Client Partner providing project support and this resonates. Two years in, the program is so f’ed, morale is so low, the project plan and execution is so terrible, it’s inconceivable how a company can exist with so much pervasive dysfunction.

Go live is May. No chance that will happen. So many defects they can’t keep track. No business processing mapping or desktop procedures beyond L1. No data mapping. No requirements were ever submitted. Still processing change requests in UAT. Controls are laughable and do not meet PCAOBs criteria for completeness and accuracy. Still missing and adding reports. And a dozen other concurrent technology implementations happening in parallel with interdependent integrations that depend on this go-live, with the legacy platforms discontinuing by end of 2023 that will leave the business incapable of performing critical business processes.

None of my consultants want to be on the project. They think it’s professional suicide to be apart of a train headed off the cliff. I am still in disbelief and am hoping that somehow I will gain a unique wisdom by sticking it out and watching them pull this off. Or it will be a spectacular apocalyptic disaster that will make for a great story.

I worked as the technical half of in a sales organization selling a big data solution so I've caught a whiff of what the blog post is talking about. A handful of organizations interested in/using our solution were doing, what I gather, meaningful work. A larger number - my guess, being an outsider and I could be totally wrong, but just based on signals I caught from the people at these organizations I was working with - were not. The ones which were not were those places where there's some sort of data arm or data scientist team with no clear mission and were more than willing to sink hundreds of thousands to millions of dollars into big data solutions on a short sales cycle.

The industries where I felt that data was being used tended to have data as their core competency. The companies I worked with that were probably doing something useful were in the pharmaceutical, gaming, and data provider spaces. (Okay, so gaming space core competency is not big data, but their data team handled everything associated with microtransactions and other sales.)

I also worked for a short time at a PAC-adjacent company working with voter and demographic data. Because that company's core competency was data, the work was meaningful - but it is extremely important that if you do that sort of work your politics match.

I wonder if OP has considered the possibility that the work produced is not valued because the target audience doesn't want the work to be of value?

If the ML/data science work is good, then it will provide guidance for making decisions about the business to management. If it's great, it will make predictions better than the people in management. Pretty soon, someone will realize that if the data model too good, there'll be no need for layers of management. The people actually running the company will just use the reports to steer and dispense with all the supporting analytics people.

It's no wonder OP's output is mostly ignored except for signaling. The people can pretend to look at, but still make decisions and recommendations based on their gut instinct, common sense, or whatever squishy justification they want, and make themselves look necessary. If the outcomes are bad, they can just blame The Model.

So the moral panic about AI putting people out of jobs isn't coming from the people working on the line, it's coming from middle to upper management realizing that their role is absolutely useless when data science can make data-driven managements become real.

Who'll be left? The top 1%, as usual, extracting value from data and the people who create it.

I've worked as a data analyst for a regulatory agency, and while the pay was lower than many "bullshit" data jobs, the tasks work itself actually resulted in real-world results.

Then again, you can't play fast and loose when your work results in people potentially losing business, getting hefty fines, and criminal persecution. You have to approach it with the utmost respect, and have solid scientific integrity.

I'm somewhat surprised at the level of cynicism here. "Data work" is basically science: trying to understand things empirically, building models, testing them, using them to achieve goals.

That should explain it. Yes, most of the time spent doing science is necessarily a waste of time. You can't just "go and do" something if you don't know how or what it is you want to do in the first place. That's very different though from saying that one shouldn't do science because of that.

A cynic may accept this and maintain the position that a lot of "data work" shouldn't be done, that it only plays a political role and isn't aimed at understanding in the first place, etc. However, that's a fully general argument against ever trying to understand or improve anything. It doesn't hold in general, only points at ways in which the system may be improved. You may complain about lawyers, call them goons doing "bullshit jobs", but I strongly suspect you would stop complaining about the legal system if the thugs came to beat you up every weak you didn't pay them protection money.

> It's the despair that comes from knowing that we spend most of our time producing nothing of value.

I feel like this is true for IT in general. Projects often fail or don't deliver on their promises.

Years of my work either never reached production or were scrapped and re-written because some key requirement changed.

For this reason I appreciate time spent with my family - it's always rewarding one way or another.

I have learned that sometimes it is better to wait work out and see how it plays out.

To some I'm sure it looks like laziness, but I consider it more weaponized procrastination. With experience I've become better at it.

Sometimes I get that gut feeling about a project, and just can't bring myself to get started. Some basic research happens, but I spend close to zero actual time on it, sticking to other things that matter.

In two weeks, priorities have changed and all of what I was asked for has now changed or been cancelled anyway. Glad I didn't get too invested.

I think you can say the same about most professions that are "big picture" ("individual picture" professions meaning teachers, doctors, EMS; folks that help individuals rather than larger trends).

The reason here is similar to why science oftentimes appears fundamentally worthless - most of the time you uncover absolutely nothing of importance. I think the same is true for data organizations. When you're looking for large patterns in the data you're going to strike out a lot. And that's not on an individual level, but on an org level - a lot of teams within the broader data organization will never produce anything of value. But at least they'll try.

With respect to data quality - fix it then! That seems like an immediate way to make your work less "fundamentally worthless". I understand that fixing data quality is hard (I spent >12 months working on that problem at my current company), but it is not a worthless endeavor.

Depends massively on the company! I am indeed one of the nerds that goes to bat for fixing data quality issues, but it takes long enough that I frequently look at these projects and think:

"It might take the majority of my working life to help this one company migrate this terrible enterprise system to a much better enterprise system. All my time will be spent in meetings, and educating hostile people who just want to do the work faster. Is that something I'd be proud of at the end of my career?"

And the answer is, to some degree! But I think we'd all like to work with people who get it or our friends, and if we're not there yet, it's really a question of whether we're at a position in our lives where we can search or prioritize other life improvements.

> "The institution produced a fair amount of academic research, but the vast majority of it was essentially fraudulent."

Wow!

Sadly that seems to be the norm. It stems from a culture where "research" gets abused to support preconceived ideologies. It should be the exact opposite way, organizations forming their ideology based on proper research.

This was true at my private, for-profit employer too. I have since changed jobs, as I was getting severely depressed from having to continually fight with my boss (the Chief Research Officer) about being unwilling to "fudge" results.

"There are usually at least a few tasks required for the organisation to function, such as producing some sort of report for the government, that technically does need to happen, so we can't simply lay everyone off." - At highly regulated companies and companies of sufficient size, reporting to upper management, government and external stakeholders is more than just "a few tasks". This, alongside a continuously developing product ("terrible data"), product pivots ("vision is hard") and growing data volumes IS what "most" data work is, and it is not fundamentally worthless.

So the premise of the post seems a bit flawed. The author seems a bit jaded by bureaucracy and lack of vision at a few organizations they have worked at, which is unfortunate, but by no means an accurate representation of data work.

That's ignoring that much of that government mandated regulation is worthless in itself, not producing the originally intended results and only leading to useless jobs like the author's.

The data is adjusted so that it fits the regulatory demand, not the other way around. In many cases it's still pointless work that produces no value for society.

Yes, I considered raising this point. My work is sometimes necessary to comply with regulation, but frequently that regulation is inherently pointless. That said, this seems to be a different category of worthless. My regular work can literally not be done. This category of work must be done or you risk terrible consequences, but it probably shouldn't be done.

I had a job like that once and I had to bail. It was essentially free money but I could barely make myself get up in the morning. I despise busy work, and if I am spending time making things that nobody is using then busy work is exactly what it is. Either give me something useful to do or give me the day off.

Amen, it's so rough. I couldn't understand why my health deteriorated over Covid, because surely not having to really do any work from 9-5 was a good thing! Turns out it's like pumping a mild poison straight into your brain over several months.

Places where data is the product are categorically different experiences to places where data is ancillary. Having a direct link to revenue or cost makes your team and your work self justifying, which is good for meaning, for influencing decisions, and for job security. It doesn't even have to be the product of the whole company - a financial risk org is a good example.

If you've been brought in as a Data Whizzkid to do Cool Data Stuff, then your job is Sales. Or you could call it consulting, but that's just Sales too. You have double the work because you need to constantly be identifying good problems to solve; solving the problems; AND convincing people that they should adjust what they're doing to use yoru work, which is always an uphill battle. My guess is that OP enjoys #2 but hasn't done enough of #1 or #3. Which is possibly not their fault: for reasons others have outlined, many businesses are set up to make #1 and #3 impossible.

My anecdotal and situational advice: if you don't like doing the sales work, do everything you can to turn your job from Consulting DS to Product DS. Find the place where your insights are most valuable to sales or revenue, and productize it -- make it repeatable, build processes in other teams to operationalize it, automate those processes, measure the outcomes, repeat.

If that's not possible then accept that you're in Sales, and that's ok, even if you need to spend way more of your time on sales than "the real work". Organizations are _hard_, arguably the hardest problem we face as humanity right now. We aren't entitled to them being perfect, we're all creators of friction as well as victims ("you are traffic"), and if changing processes and agitating is what's needed to make your work valuable then that's what you actually got hired to do - regardless of what your job description says. (Not trying to say it's possible or worth it in in all or even most situations like the ones OP describes, but it's worth a look before becoming resigned or quitting.)

I've been working in the data space for 8 years and find it incredibly fulfilling. Open source community is great (I've built and contributed to a lot of projects), LinkedIn has tons of great data content, meetups are fun.

The work is interesting and the problems are important.

It's funny cause the parts the author is mentioning are easy are the hard parts for me. The author says "The pressure is non-existent"... That hasn't been the case for me. Deadlines are tight, jobs fail, running jobs overnight, checking if they're still running before going to bed - it's not great for a restful sleep.

I've always given talked, created new open source libraries, and blogged whenever my day job gets a little boring. It's such a fast growing, exciting field. I feel like it's hard to keep up, let alone get bored!

That's the funny thing - as I mentioned, it's easy to feel bad complaining because these jobs were objectively quite cushy. But you're absolutely right, giving these things up and doing something that matters entails actual stress. I'm sure that someone else could write an article talking about how deeply they wish they had work that doesn't matter, and the absurd thing about human experience is that they wouldn't be wrong at all.

I just started reading some of the sites linked on your profile and really enjoy them, by the way! Really wish it had turned up on Google the last time I was working with Databricks.

To rephrase the title a bit: most data work seems fundamentally like research.

You look at a bunch of data seeking a nice, clean X/Y relationship that yields positive ROI for the org. These types of situations are rare - that’s the sort of e-commerce landing page / conversion optimization problem where a bunch of other factors (particularly whether anyone needs or wants the widget) are answered already.

That doesn’t sound like the nature of these data engineering type roles. Sounds like there’s an explicit “Get us this spreadsheet!” BI-grunt functionality, with the overhead of doing researchy type work that might pay off for the company later.

I’d guess that the “get the business unit their spreadsheets” is the actual valuable part of this role, and any other useful insights about the data are just considered gravy by the org.

This is the dirty secret of the industry. Companies hire data roles without knowing what to do with them.

It's not that data work is worthless. It's that most people aren't very good at trying to figure out how to tie data to actionable events that drive business value.

Bad news - It's not the business' fault - it's your fault. If you're a data practitioner and you feel this way about your job, learn the business and seek to find opportunities. If you just do the work you're told to do, you probably won't magically create value.

Good news - There's a lot of opportunity to help educate data practitioners how to think about data through the lens of business problems. I'm personally trying to tackle it through developing an online course.

This is typical of the public sector, lots of money from tax payers and over spending with no accountability.

No wonder why we can't keep our debt under control.

I have many friends and acquaintances working for the federal government in different sectors, from DoD, to education, housing and health and the stories are similar, outrageous budgets that go to waste and practices that sound borderline ilegal.

There's no common goal or long term vision and everyone is aiming at their promotion to the next pay bracket.

For service providers there's also no incentive to be efficient and deliver the best results because public contracts are not necessarily assigned to the most capable organizations, all you need is the right connections and once you are in you keep getting projects.

I don’t think as an industry we’ve actually solved the problem and started to deliver on the potential of data yet.

Most companies are hacking around with messy stale data, excel sheets, dashboards and doing rudimentary analysis. This is essential level zero on the maturity scale, it’s like rubbing sticks together to try and spark a fire.

Nowadays we know what good looks like. Data engineering, online machine learning, collaborative analytics environments, embedded analytics, streaming etc. We just need to get there.

My impression however is that the people at end clients are a bit stuck in a local optima. They think ETL, data warehouses and dashboards when what they actually need a total rethink and change in approach.

Someone I know works on catastrophe modeling for the insurance industry. It seems like a fascinating area, especially since there's a strong link with climate science - a large part of the business seems to be modeling hurricane risk in Florida and wildfire risk in California. The complexity is because these are fairly rare, highly correlated risks - unlike e.g. auto insurance you can't just take a few years worth of claim data and predict your losses for next year, unless you want to go insolvent the next time a sufficiently big hurricane goes through a sufficiently prosperous part of Florida.

I switched out of data engineering, and became a software engineer instead. I work on building the data infrastrue tools that I used to use. Maybe this only applies to me, but building the tools that I used to use is somehow incredibly satisfiying. I think it's because as someone who used to use the tools, I know what I want, and then I build it, completing the "cycle of needs".

So my advice to some of the data scientist/engineers out there is to go a little deeper into the tooling and try to understand how they were built.

The second was my barber. He's an immigrant here, and started a very successful business. Unlike the young tech entrepreneurs I typically meet (who glorify money and the wealthy in a way I find grotesque), he's a very quiet man who has incredibly happy employees and remembers every customer by name.

Refreshing to hear this. For a long time, there was a persistent view in tech startup land that anything not designed to scale fast and cash out 10x for investors was fundamentally worthless, and was typically disparaged by the VC class as a mere "lifestyle business."

Let's take one step back and have a look. The main cause of the output produced being bad is... bad input. Find a data science job where you are part of producing that input by gathering raw data yourself or at least have an impact on how it gets measured. The first datasets will be garbage but at least you have a feedback loop that tells you how bad, in what way the data could be wrong. You can bring in theories, read papers,... to improve (the measurements of) the data. You will see gradually your data starting to make sense. Wouldn't that be fulfilling?

"Data science" related work is absolutely fundamental for fixing many of the world's sustainability problems and also creating conditions for more enjoyable and fulfilled lives. Alas, we have political and economic systems that misallocate human talent at massive scale. This is not new or unusual. Heck, human life itself was never worth much when it came to power struggles between various oligarchies.

People with a gift to work productively at the interface between digital machines and society should feel privileged and act on it rather than lament about bullshit jobs.

Why not try to do data work in science? Analysing astronomy data, particle physics data, or DNA sequences sounds high impact, and you have world experts on the data to make sure you’re getting quality curated data.

Absolutely an option! I've got my own strategy out, but it seemed self-absorbed to go on about it. People don't want to hear about me specifically beyond whatever minimum lived experience is required to make my point. Plus it hasn't panned out yet.

Maybe it's good to also notice that the author is mentioning "data" but talking specifically about doing PowerBI dashboards, reports, etc. Which means that they are working with "data" in its least objective and more bullshit-prone use case (I'm a visualization researcher and I can tell you, it is 99% bullshit). One could possibly cut the fat and turn into machine learning, but that'll also require a lot of extra training and a thick skin to survive in what's probably the most competitive IT market right now.

Having way too many bullshit jobs backfires , and we are not prepared for that

My SO and I both work in data (in healthcare and higher education, respectively). Neither of us has had this experience. Sure, some of our dashboards have died, but many of them have also provided real value. My SO's have literally saved lives! We each benefit from having direct contact with senior leadership in our orgs, and good understanding of the business side of our industries.

Other than that, I'm not sure what advice I can give to avoid this problem, but I'd like to assert that the worthlessness of data is not a universal experience.

Most software engineering work seems fundamentally worthless (and it is)

How is that?

A lot of software is just like a lot of white collar work - just bullshit moving stuff around.

The key word is most - A good example is many enterprise software companies.

They sell a product made to fit a purpose that someone in power thinks it should, but doesn't qualify whether or not that's the case. The customer is sold the software as an appeal to their ego and supposed needs that they might spend millions of dollars on.

The people on the ground have the software "purchased" for them and they basically are told "it's this or the highway" quarterly releases often leave people head scratching why anyone would give a shit.

Big parties are thrown to celebrate implementations that even in the medium term have no utility and you would been better off spending it on hookers and blow.

There's so much of the industry that's just running on a treadmill.

The way I look at it, is that if your job disappeared overnight would anyone even notice? A great example here is Twitter. Elon fired (by conservative estimates) over 2/3 of their total staff. Yet the service runs just as it did before. What were those people doing? Did their work matter at all?

In almost every medium or large business there are easily millions to be saved or made annually with the correct data. You're doing something wrong. Read what you've written here. You're doing a lot wrong. If you're on a garbage team, stand up a little taller and take charge. Hire your own team to do things correctly. You're unhappy because you're failing. You need to reflect on that and change or remake yourself, possibly in another field.

Often, the job of a data team is not directly to produce value. It is to support value creation through observability.

Observability gives the business side intuition and knowledge about how the business works. The fact is, no one in the organization can get visibility at scale into their operations without a data team.

Now, I have argued before that data teams can and should also focus converting knowledge into action. This moves data teams out of a passive, reactive role into a proactive, value-creating role.

It's very hard to know what creates value without more senior leadership experience.

Sometimes it's only '1 thing out of 10' but that '1 thing' is absolutley essential.

But honestly, I suggest the entire 'big data' movement is a bit outlandish, there really isn't a need for most of it. The 'saving grace' frankly is AI, because maybe-maybe-maybe we can make use of that data in some future alg because god knows we're really not making that much of use of it.

But it's hard.

A lot of times senior leadership isn’t equipped to communicate the data requirement to achieve a task. So data people, technologists, dbas, are shooting from the hip trying to fit a square into a round hole.

Once the target is hit, most of the work to hit that target is worthless, but we’ve gotten smarter at taking the journey and creating repeatable steps, setting up data structures, etc.

I also worked at a company that had antiquated data storage practices and ugly political silos around it to the point where it was kafkaesque to access basic operating data which was actually valuable.

... and were the goddam strategy team!

I really depends so much on leadership as well.

I suggest most companies either have 'Zero IT' leadership knowledge, in which case you get the 'buy whatever Microsoft tells us to buy' - or they are tech companies where they understand it but are probably a bit overzealous.

But the same thing applies to everything.

'Legal' can be a useless waste of money, until all of a sudden they're the most important team in the company ...

Anecdotally, it seems like companies advertising data + AI capacities are the worst offenders when it comes to buzzwords and grift. Also anecdotally, I know the founder of a small agency that does rapid prototyping of predictive models for logistics-heavy industries and adds a lot of value.

I think the problem is really one of scale - it's way harder to pin down your value add when you're part of a very large collective. Being responsible for 10% of a 10-man output is much more tangible than being responsible for 0.001% of a 10,000-man output, and the difference between 10% and 0% is similarly much easier to discern from a management and accountability perspective.

This.

So many people underestimate the value of experience. Seeing patterns comes with seeing many different things over an extended period of time.

I think the author wound have been very grateful if at least some of the seniors would have seen it as part of the leading task to craft some reward structure for their subordinates. Like that barber did. Life is so much nicer when money is for people, not the other way round.

The teams that brought value to the company were embedded alongside of the folks producing the data to be 'scienced' and had leadership that was able to understand and communicate the information (and its limits) to their leaders and adjacent teams and use their experience to point the data science folks in the right direction.

I know that everyone is getting into the nitty-gritty of the Data Science vocation, but I (a non-Data Scientist) see this as someone that is fundamentally dissatisfied by their career, and that is something that many of us can relate to.

It's certainly something that happened to me, and getting laid off, and then shunned by the industry, ended up being one of the best things that ever happened to me.

I've spent a good chunk of a half-century-long career, now winding down, doing the kind of work our author describes. Most of it was on my own initiative. I'd notice some kind of anomaly in the logs of whatever app I was working on, investigate it, and generate a report program highlighting it.

Most of the time people ignored this stuff. That took some getting used to.

But occasionally one of these little report programs would show a solvable problem. The usual example was a whole bunch of customers using the app in some way unpredicted by product designers and developers, and so missing out on a valuable feature of the app. Often we were able to add some kind of workflow feature, or add a UI afffordance to help customers take advantage of existing features.

(Other times the little reports detected performance trouble. Always consider using time-of-week as an x-axis when working on these little reports.)

The hard thing about doing all this is that I, and my colleagues, never knew ahead of time which data anomalies would be actionable and which were just fun facts to know and tell.

Sometime along the way, that part of my work was dubbed "data science". That's about the same time a bunch of enterprisey software entrepreneurs discovered that "dashboards" generate sales because they appeal to front-office folks with control over money. Irony: I developed a bunch of integrations for a really expensive data-analysis product using Jetbrains tools I paid for personally.

I always thought of that part of my jobs as diagnostic and exploratory. What can we learn from how this system works in the real world?

In my case, of course, I also had some responsibility for the systems generating the data I analyzed.

My advice to others doing this:

1. Always always assume you'll be called on to generate recurring reports with your little report programs. Make or buy a report-generating tool that can, at a minimum, deliver CSV files by email.

1. Indulge your curiosity. Especially with strange and incomplete data sets. Don't think of your task as "generating a report from bogus data for people who don't give a s**". Think of it as "figuring out how to make sense of the process from the data it captures as it operates".

2. Learn all you can about the processes you're analyzing, be they failed logins to SaaS apps or ambulance-calls to wrong addresses, or whatever.

2. When you have nothing much left to learn at a particular employer, teach somebody else how to do your job and then move on.

I can't speak to "data" jobs in particular, but I have had programming jobs in my career where I felt like the OP, and ones where I didn't. If it is important to you that your work be valuable in some way, I suggest you not give up. Do what you can to find work that is. It is out there, maybe you have to better yourself to raise the odds, so try to do that if you can.

Most science is fundamentally worthless. We're not accustomed to that in a business setting. 99% of science yields no interesting result.

The only decent data job I ever had was analyzing time series data from war planes. I say decent because we had the data needed and clear goals about what we needed to achieve. Something real and tangible.

Then it turns out I'm directly helping war criminals (Saudi Arabia) bomb mothers and their children. Had to hit "eject" as soon as possible.

I mean, you wrote code for a weapon system. Did you think only “bad guys” were going to get killed?

Weapon systems are banned at GE. It takes an army of contractors in the private sector to keep fighter planes working (any modern military hardware, really). Operate them in a sandpit and they last about one day. Unexpected failures routinely occur and numerous data scientists, data engineers, and aviation engineers get spun up to figure out what happened ASAP.

No, I did not expect to be helping warlords. The US is stuck in a really super-shitty policy with the Saudis right now. Contractors (defense firms) have zero choice in supporting them unless they are sanctioned. Meanwhile the Saudis have been acting unilaterally (bombing whomever without consulting the President) for about fifteen years now.

My data work saved one of my clients in one of the project probably about 2M for the price of <500k - and enabled revenue streams of a few M.

Seeing a bunch of opportunities of similar proportions. It's ugly work, but my theory is that this it why it pays.

I disagree with the assertion that quality and meaning are somehow related. Paradoxically, it is often the worst code that is the most meaningful in terms of its impact to the customer and business.

If your product (data, dashboards) has no users, then it has no flaws, and it is perfect.

Put in the absolute bare minimum of effort and move on to more interesting things.

most data work is fundamentally worthless

future knowledge workers are going to look back on the 80s-20s the same way we look at miners using hand pickaxe and shovels

While I work in data and do appreciate the sentiments in some ways, I'm not sure this feeling is unique to any type of office work?

I don't really understand the despair. The only thing I can think of is they somehow have their self-worth tied up in their work. For me, work is merely a way to earn money for those things that actually matter to me - family, hobbies, volunteer work, etc. My actual work has no real benefit to society. As long as they keep paying me, I don't really care. That's on them to want me to work on things that fundamentally don't matter.

with a six figures job and leaving it at 5PM, you still have a lot of resources most people don't to make something impactful. I think you're just trying to justify your inability to do something meaningful with other people bad decisions. Sorry if I seem harsh, but that's how I see it.

I thought this for a while too. Nothing wrong with calling it like you see it. Suffice it to say that I've seen a lot of people try (and even succeed partially) to improve things, but all of them would be producing stuff ten times better if they struck out on their own or found a better organisation.

In that brief government stint, I realized there are very good reasons that a few well-meaning and smart people haven't just fixed the government. It is a horrendously difficult problem.

ethics is probably the hardest problem out there

loved this. when you say

> They also can't code - the degree to which my teams sucked was basically directly correlated to how good my manager's people skills were _and_ whether they had programming experience

do you mean your manager knowing how to code correlated to a good or bad team?

Good question, I should have been clearer. I've heard that managers who can program can do some horrific micromanagement. To be clearer, when I've had a manager who both had good people skills and could program, life was pretty good overall. Good people skills alone was still okay in some situations, especially if they at least learned the basics of what our tech stack did conceptually. I've never had a manager who could program but had awful people skills, and from the stories, that's for the best.

Most software is. I'd say 1% of what I've been paid to code was useful to humanity.

I know of no better life purpose than to perish in attempting the great and the impossible.

This is the best thing I read in months. People should read David Graeber’s (in memoriam) Bullshit Jobs.

People are great at fooling themselves but on the other hand we often instinctively know when our jobs are bullshit.

And I think 30%-50% of IT jobs, it being operations, software development or supporting roles, it doesn’t matter.

It’s a combination of incompetence of people and the incentives of capitalism.

Thank you! And when pressed, almost everyone I've spoken to that hasn't moved into management (if you ask management, you're asking for people who have largely self-selected into delusion, at least at these orgs that are struggling), has admitted their work is useless.

I met an engineer considerably more senior than myself recently, who said something along the lines of "A reckoning is coming". I think they have a lot more faith than I do in these ultra-bloated organisations trimming fat. The funny thing is, we both think that we're the people who should get trimmed, not for lack of trying, but because our best efforts have yielded nothing. I keep getting promoted, but if you look at my track record, almost everyone I've ever worked with including the super smart and diligent, are abject failures.

Yes, I agree 100%. The reckoning isn’t really coming because too many people in positions of power - who often care about headcount - and so many people don’t have the insight / knowledge that the IT team can be 50% of what it currently is.

This sounds like most of my dev jobs to be quite honest with you.

A lot of the problem has to do with the entire approach of these organizations: tinkering at the margins of neoliberalism. Very little work gets at the real problems of our political economy. But that’s by design.

Definitely not just data work either.

This platform looks refreshing.

It's the only thing I can write on, because it is snappy and I have nothing to focus on other than writing.

After a decade of referring to "data engineering" by asking candidates, "So you want to be a plumber?", this resonates.

My dad wasn't a plumber though he was a member of the steamfitter's union at one point.

These are similar problems though they may not look alike at a casual glance:

Plumbers make good money, but I'm not sure there's deep meaning in the pipes because, like metadata, pipes go sideways too.

I'm having trouble thinking of a Turing award winner who won because they went sideways. The key is that the data insights of a Turing award winner are durable, unlike the transient nature of most of what we do with data in modern organizations.

Ultimately, this sounds more like Grothendieck in math than computer science:

https://www.psychologytoday.com/intl/articles/201707/the-mad...

Metadata sounds like it's the wrong word as the structure we are looking for isn't the ephemeral structure that most of us discover, but rather, it's closer to special relativity.

The Turing award winner who comes to mind is Leslie Lamport:

https://en.wikipedia.org/wiki/Logical_clock

There may be a better choice that demonstrates the worthlessness paradox of data engineering or metadata or data science in a modern management context.

It's not that metadata are actually worthless as in fitness-for-a-particular-purpose. Rather, it's that the insights that we and our organizations reap from that metadata is increasingly short-term - anything but durable.

https://plus.maths.org/content/stuff-happens-ordering-histor...

The only references I can think of on quality are either Pirsig's MOQ:

https://en.wikipedia.org/wiki/Robert_M._Pirsig

or Wallace's posthumous Pale King:

https://en.wikipedia.org/wiki/The_Pale_King

Your friend's experience working in government sounds exactly like the Pale King to me.

The worthlessness you're referring to seems much more dependent on the fact that management culture has evolved to somewhere between a cargo cult and celebrating short-term things that don't count, both illuminating an illusion of control.

I mean, does anybody, anywhere still get a gold Rolex and actually retire?

Basically all engineering job is kind of... pointless circa 2011 if you belong to working class. The novelty value of engineering is gone. We see where it's all going now. We won't enjoy the fruit we help grow. The protest of 1968 was right about everything. The boomer We're here to repeat the same mistake. Modernism and endless worship for overproduction. Overproduction of goods led humanity into two world wars and now we have overproduced information on top of all that. What's even going to happen? What's this all for if we can't stop the climate change anyway? But don't lose hope. Ai is the last hope sort of... or the worst thing.

Reply on news.ycombinator.com