Comment by munchbunny | Hacker News Reader

Comment by munchbunny | Hacker News Reader

Parent Full thread

As a data engineer in big tech, the two hardest problems I deal with are:

* Conway's law causing multiple different data science toolchains, different philosophies on model training, data handling, schema and protocol, data retention policies, etc.

* Coming up with tech solutions to try to mitigate the impact of multiple silos insisting on doing things their own way while also insisting that other silos do it their way because they need to access other silos' data.

And the reason standardization won't happen: the feudal lords of each of those branches of the hierarchy strongly believe their way is the only way that can meet their business/tech needs. As someone who gets to see all of those approaches - most of their approaches are both valid and flawed and often not in the way their leaders think. A few are "it's not going to work" levels of flawed as a result of an architect or leadership lacking operating experience.

So yeah, it might look like technical problems on the surface, but it's really people problems.

I can add so many:

- Requirements are rarely clear from the beginning;

- We (DE) are not enabling self-service and automation so we are drowned in small requests (add this column for example;

- Upstream rarely notify us about the changes so we only know when downstream alerts us. We end up building expensive pipelines to scan and send alerts. Sometimes the cost of alerts > cost of pipeline itself;

- We have so many ad-hoc requests that sprint is meaningless. If I were the manager I'd abolish sprint completely;

- Shadow knowledge that no one bothered to write down. I tried to write down as much as possible, but there are always more unknowns than knowns;

Working in DE definitely gives me enough motivation to teach myself about lower level CS.

What does DE mean in this context?

Presumably Data Engineer.

data engineering

data engineering

Divest and exclude? …

Wrong answers only please.

That's the mother of all people-space problems in IT, right there.

To solve this, one can be an instrument for change. Network, band people together, evangelize better ways forward, all while not angering management by operating transparently.

Sometimes, that can work... up to a point. To broadcast real change, quickly, you really need anyone managing all the stakeholders to lead the charge and/or delegate a person or people to get it done. So the behavior of directors and VPs counts a lot for both the problem and the solution. It's not impossible to manage up into that state with a lot of talking and lobbying, but it's also not easy.

I'll add that technological transformation of the workplace is so hard to do, Amazon published a guide on how to do this for AWS. As a blueprint for doing this insanely hard task, I think it holds up as a way to implement just about any level of tech change. It also hammers home the idea that you need backing and buy-in from key players in the workforce before everyone else will follow. https://docs.aws.amazon.com/prescriptive-guidance/latest/clo...

> It also hammers home the idea that you need backing and buy-in from key players in the workforce before everyone else will follow.

Yup, this is the key issue and what makes it primarily a people problem. Technical solutions don't work if the main problem is getting buy-in to spec/build/adopt one, unless you're willing to build a lot of things you end up throwing out. So instead the bulk of the high risk work is actually negotiation between people.

> And the reason standardization won't happen: the feudal lords of each of those branches of the hierarchy strongly believe their way is the only way that can meet their business/tech needs

I work in implementation of large enterprise wide systems. When I do projects that span departments/divisions/agencies what you’re describing is the biggest hurdle. The project always starts with “we’re bringing everyone together into one solution” but as time goes on it starts to diverge. It’s so easy to end up with a project per department vs one project for all. You have to have someone with the authority to force/threaten/manipulate all the players onto the same page. It’s so easy to give in to one groups specific requirements and then you’ve opened Pandora’s box as word spreads. It’s very hard to pull off.

I think public sector (governments) is the hardest because the agencies seem to sincerely hate each other. I’ve been in requirements gathering meetings where people refused to join because someone they didn’t like was on the invite. At least in a for profit company the common denominator for everyone is keeping their job.

Reply on news.ycombinator.com