Hacker News Reader

This is a lesson I keep repeating to our juniors: when you do something notable, celebrate it, and make sure to include the whole team, the users and the boss in the celebration.

It can be as simple as an email, but sometimes, making a few chat messages or even calls. It takes times, it seems silly, but you get a fantastic ROI on this.

Because now people see your work as valuable. It's concrete to them. They understand its value. And this has many positive consequences on your life.

This, of course, works only if the celebration avoids technical terms, is short, straight to the point, and explain what's the benefit to them and the org and how much it took to to it.

People must know that you are doing a good job, and you are the only one that is in the position to let them know. They will not get out of their way to learn about this.

E.G of a chat message I would send to my client's team boss:

Hello <first name>, it took 3 weeks, but we finally finished improving <system x>. This should save us <y> of <z> in the future. We tried hard to make sure your work is not disturbed by the change so it should be transparent for you. But let me know if you notice anything out of the ordinary. Cheers

Believe it or not, people actually like it. It keeps them in the loop, give them context and perspective, and help them take better decisions. So don't hesitate to do it.

> It keeps them in the loop, give them context and perspective, and help them take better decisions

Other side of the coin: "I just spent 2 weeks optimising a daily process to be twice as fast" might well end up in me wondering why anyone would care about the 1 minute reduction in delivery time for daily cadence data. It might also result in me being stroppy about work I cannot get resourced that has obvious $ value.

But that is also a win for the business. Either I am wrong and am persuaded of the value, or I am not and I become more engaged with stakeholder prioritisation and we set up something more effective.

That's good feedback to have. If your boss/client step in and says "we don't need that", this allows you to re-calibrate the team priorities and shows you need to get better at extracting the real need in the future.

I completely agree with your initial comment and your response here. Transparency is a great feedback loop. And it's always good to celebrate what you do.

Or just do this

1) Break something on purpose. 2) Make sure the business notice it (usually a spectacular cost increase does the job). 3) Come in and fix the issue 4) Make a cost comparison quick chart before/after 5) Become famous

You don't need to do it artificially.

If you look long enough in most code bases, you will eventually find a spectacular problem.

Case in point, I was working for several months for a client, and I had a slow afternoon, so I decided to convert some calculation to numpy, see if we gained any free perf.

We got a x100 local speed up, which was very fishy. Gaining speed is common with vectorization, but two orders of magnitude is a lot.

So I looked at the original algo. There was a glaring mistake in it, that I fixed by my numpy code without noticing, and just changing that in the pure Python algo, without using numpy at all, made it X50 faster.

I could then call my client, and celebrate the good news. Not "there was a mistake", no. But "we found margin for progress".

>I could then call my client, and celebrate the good news. Not "there was a mistake", no. But "we found margin for progress".

This is great framing.

Bugs in code are not really "mistakes"; they will occur. Finding and fixing them is a positive.

> Not "there was a mistake", no. But "we found margin for progress".

Were your clients nontechnical? Or did they just not care to know how you got a 50x speedup?

The boss only cared about the end result and got a quick message. He won't read more anyway.

The technical team an in depth explanation with code snippets.

Know your audience.

This hits home. I communicate a lot more with up and downstream stakeholders as a data engineer. Be it for designing future changes and how those affect others. Or, as you said, triaging outages/delays, figuring out where the missing dependency is. Less problem solving by coding.

The SQL thing the article mentions is also true. Even if you work with higher level tech like Spark. SQL gives you a good feel for joins. I also like the fast feedback cycles you get from SQL compared to something Scala/Python based.

This is exactly why I found DE is not a long term career path. I work in data, I occasionally do DE roles, but if you aren't moving over to the app side or business/domain side, you are leaving a lot of money on the table long term.

The funny thing, being in data, if you take the time to understand it.. you are tremendously valuable to the business/domain side.

Otherwise you 1000% risk becoming part of the background infra. No one celebrates when the faucet dispenses water or the power switch works. They do lose their collective #$#@ when the water or power fails though. Data Engineering has a tendency to be treated that way.

DQ is a big part of it as well. Everyone wants it, no one wants to pay the hours/dollars to do it.. No one is sure what they want, but probably what you propose is too simple. Can't AI do it? Etc.

TBF, in most companies that have a traditional core business (let's say, selling pharma products), IT is a cost center and your run of the mill engineer is seen as the plumber who's suppose to make everything seamless, as you describe.

Short of solving a pain point in a spectacular way, the whole IT dept will be the face of stuff crashing, security issues and budget burning.

> if you do an absolutely superb job you end up being largely invisible to the business

openly communicate how difficult some feature, project or solution is going to be. estimate time then multiply with pi and round up. then proudly present how you got it done despite obstacles possibly even in less time.

also good - create problems - then solve them.

The Futurama quote is relevant: "When you do things well, nobody will know you did anything at all". I think this goes for a lot of engineering, though. Especially infrastructure.

> Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected.

Sounds a lot like Cybersecurity.

Cybersecurity jobs get paid very well, though.