Ask HN: Shift from Product to Data Engineering
1- sources to learn
2- does my background help or hurt?
3- best way to break in
4- any "career therapy" sources
--
I have a PhD in chemistry and fell into PMing 8 years ago. I have touched a lot of technologies and dealt with embedded systems, consumer tech, SaaS and, of course, data analysis totally out of my field and comfort zone.
My current company is good enough on the surface- good package, cool clients, cool tech. It does suffer from key drawbacks - nepotism, micromanagement, silos, total lack of strategic awareness. The result being old-timers rule the roost and even though we're 500 employees it's still managed like a 15-body startup. Turn over is high - senior hires tend to leave in a hurry (1-3 years). I was brought in to help with one of the main product lines. The idea of the hiring manager to get a senior with outside experience to scale. The hiring manager resigned a few months later, so my role was redefined lackey, writing content I don't think is useful in any way. I stuck around past the point of it being obvious that this is not a good fit because the pay is decent but things went really sideways as I've fallen hard with the old-timers (everyone else loves working with me, to the admission of one of the execs who added "I don't understand why"). So I'm being put on a PIP (first one so it stings). Even if it works out, it's clear that I'm better off moving on. But it occurs to me I might not actually be suited to PM and have never lasted more than 3 years in any PM role. As I look back at my career, there are always good justifications for my moving on from previous posts. For one thing, every hiring manager was gone within months of my start date, and there were other reasonable excuses (e.g., pandemic raise, acquisition, role totally redefined, etc.).
Still I am going through an exercise of self-evaluation to understand why I don't seem to find satisfaction in the roles and why I jump.
Part of that is an understanding of my own character (is my ego too big, do I expect too much from leadership). But the other part is the career choice itself. I got into PMing because it was sold to me as acting as the liaison between engineering and sales with P&L. So your levers were VOC to engineering and sales strategy. It worked well enough. But now, it's common to see full ownership of everything, from architecture to P&L without formal authority. So you need to be a technical expert with deep business development, strategic and tactical capabilities, who engineers and sales reps will follow because reasons... So by its nature there is a lot of strong-arming and conflict. I really don't have the disposition for that stuff and it causes immense stress (aside- this also tells me I won't make a good CEO).
So I'm thinking instead of jumping into another product role, I'd be better working on something "real" and this takes me to data engineering. Like I said, I have a PhD so I'm good at analysis and basic programming with exposure to deep learning and other advanced statistical techniques - definitely not an expert. Data has always been important for me, mostly on the quality, validation, analysis and visualization side but also exploring and understanding the pipelines. In my spare time I have fun in Python and Pandas for several years.
I would like to formalize and focus this hobby to learn data engineering for my own benefit, but also see if I have the chops to make it a viable option. On-the-job training won't be possible given the circumstances, and it's contracted out anyway.
So I'd like to ask:
1- sources for self-learning data engineering?
2- is my prior experience a help or hindrance?
3- the best way to break in?
4- any places to test or benchmark my current skills?
5- any sources for general "career therapy"?