Comment by whack | Hacker News Reader

Comment by whack | Hacker News Reader

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset.

I sometimes gave my own suggestions on how to improve upon their ideas, but for the most part, I was happy to focus on implementing their ideas, in the most clean, elegant, robust and testable manner possible. I was happy to do the "plumbing" work of improving upon our tech stack and architecture, in order to make the entire system better functioning and easier to maintain.

According to the author, I'm supposed to resent the fact that I'm a "doer/plumber", and not a "thinker". In reality, it was the opposite. Do I really want to spend my entire day reading the Bloomberg manual and figuring out which tables/columns will give us the data we want, and the nuances of what this dataset does and does not cover? Sorry, I have zero interest in doing that.

I enjoy programming. I enjoy system design. I enjoy building stuff. I have zero interest in becoming an expert on how to interpret the Bloomberg symbology file. Besides, if I ever left the financial industry and joined a tech company, that knowledge will become completely useless.

Did I or anyone consider myself to be a "menial" plumber? I don't think so. I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated that I could quickly and robustly implement the ideas that they had, and keep the system running smoothly without hiccups. They recognized anyone can do a "good enough" job, but it's much much harder to find someone who can do a great job. And for my part, I was perfectly happy to be that guy.

If you're someone who wants to expand your breadth and take on more "thinker" responsibilities, more power to you. But just don't forget that there are people like me out there too. There's no shame in being an excellent "doer".

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything but spewing out whatever pops into their head. Half is nonsense, one-quarter would be actively destructive if you tried to implement it, they always seem to know everything about everything but whenever it’s something you know really well you can tell that they are very confused, etc. when I meet these people now I just think “oh, you’re one of those guys who is good at saying a lot of things” and then move on. Oh well

Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack.

One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.

This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as many corner cases and implementation really produce high quality work. It’s all about discipline indeed.

"At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week."

Wow, true system engineering! Your company sounds like a good place to do some professional work.

Feel free to visit oscar career page.

That's great. A design doc is even a step beyond just writing down the idea. Bezos is famous for making people write one pagers to pass out prior to talking about any idea. I see one pagers as way to quickly weed through a bunch of ideas, and a design doc as the next step to determine feasibility.

It really is a great tool for validate ideas.

Sorry for my grammar above as I was typing on my mobile phone halfway on the bus.

Yes, those are talkers masquerading as thinkers. Not all talkers are bad, in fact some are great to work with, but the bad ones seem unable to not burn bridges.

That is also very true. I worked with someone like that who was really wonderful (and a wonderful person)

I once worked with a "Data Scientist" at a hedge fund that was clearly pattern matching whatever problem you had to some random Apache/Google tool without actually listening to the problem.

  {
    "Unit test" : "google mock",
    "CSV" : "Apache Parquet",
    "message" : "google protobuf",
    "queue" : "Apache Kafka",
  }

etc

His data science recommendations looked like a markov chain of various analysis algorithms.

One time I started digging into his recommendation trying to figure out why it was even on topic and he starts going on about how he's a "big picture" guy and not to bother him with implementation details. The thing was, his 'big picture' ETL was breaking our trading system every other week due to some inane dependency strictness that wasn't necessary.

There's nothing more 'big picture' than not fucking trading!

I guess because portfolio managers aren't specialized in engineering, we see a lot more of these fakers in finance than tech.

You see that a lot with 2e people -- bright people with some weaknesses or a disability. That could account for how negatively you have experienced this. Many 2e people have never really been taught good ways to handle the combination of big strengths and big weaknesses.

I serve as a sounding board a lot for my oldest son and that works well, but it's not uncommon for such people to just be trying to meet their own need to process information and/or feed their ego, oblivious to how it impacts other people and not really welcoming of the feedback they really need for this to be constructive. A good sounding board doesn't just listen, they ask pertinent questions and make insightful comments that help move the thought process along.

Sometimes when I meet people like that, I'm able to direct the conversation to a more constructive back and forth of that sort. But some people just know they have this need to talk, they have a lot of baggage that makes them openly hostile to meaningful feedback and they crave validation. Anything other than praising their half-baked ideas is met with toxic reactions. In such cases, the best you may be able to do is basically make a few polite noises and then disengage as quickly as possible.

What does "2e people" mean in this context ... That's not a term I've run across before?

Twice exceptional.

https://en.m.wikipedia.org/wiki/Twice_exceptional

(mobile link! D; )

To be a little blunt about it,

> they have a lot of baggage that makes them openly hostile to meaningful feedback and they crave validation. Anything other than praising their half-baked ideas is met with toxic reactions. In such cases, the best you may be able to do is basically make a few polite noises and then disengage as quickly as possible.

would serve pretty well as a fair description of normal people, ime.

True, almost everyone has baggage. I think the difference is one of scale.

For example, in grade 5 I got a C+ in fine arts. It devastated me to the point where I questioned my abilities and disengaged from schooling. Permanently. It's only been in the last few years that I've actually been able to apply myself to anything.

Now, I can see that was an unreasonable reaction. At the time though, I didn't even understand what was happening.

If I hadn't got a lot of help, I believe I would still be driven by my insecurities to this day and would be impossible to work with.

Edit: typo

> What’s funny to me is how many incompetent “thinkers” appear in meetings.

My thought as well. I also worked in hedge funds for a long time, and I kept getting resistance from the self-proclaimed "thinkers" to do even the most basic project management things like keeping a shared list of bugs, using version control, etc. It became clear that they simply didn't know how to use these tools, while claiming to specialize in financial modelling.

This also turned out to be false, as moving on to other funds I discovered their way of seeing things was quite limited. Which I had a good suspicion of, but other people actually showed me how one could approach things.

Part of that reckoning was that to be good at building financial strategies -something virtually nobody will tell you about- you need to be fairly good with writing code. Not just your Frankenstein of VBA, Excel, and Matlab, but including a fairly deep understanding of algorithms as well as common DevOps tools.

You mean the type of people that rule the world?

I'm joking (sort of) but this is half the job description for leadership jobs, politicians, executives and such.

The differences are of quality, not type. Jobs' thinkering was good, for the most part, for example.

I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of things. You can only lead a horse to water so many times before recognizing that they just will not drink the water themselves - some people actively refuse to implement solutions, no matter how convenient the building process might be. And then the more you burden them with things like SLAs, performance, etc., the more of shit show things become.

There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.

I've experienced that hypothetical perfect world, it comprised teams of data scientists, developers, and system engineers working on making progress towards multiple distinct 'business insights' and operating within a big data ecosystem. People's job roles were tightly defined but their participation in tasks was largely determined through self-selection and reputation.

I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.

They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.

Really? I had one sort of similar because I took as many cross listed math electives as possible and avoided any software type courses, but even then I learned a lot less analysis, algebra, geometry than a good math student would learn

Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.

“ETL” also involves data modeling. This is a thinking role beyond many people’s understanding. Quite often the “Thinkers” demand data but have very little understanding how to structure it when you get temporal aspects (slow changing dimensions etc), new types of data (rules for types) etc. These and more are quite often left/relegated to ETL guys as housekeeping. Except it isn’t. It’s architecting the house.

I was once accused by a Windows C++ programmer who just couldn't grok UNIX to be a "string cutter", but honestly, there is nothing so viscerally information technology as extract-transform-load. The irony of his accusation was that it was just one of many facets of things that I do and can/know how to do. ETL is really the core of IT by definition.

A lot of the hardcore C++ programmers look down on the whole IT area of programming (as opposed to systems, graphics, embedded etc.), maybe that was the reason.

No, this guy doesn't know anything but Windows.

Completely agree, especially as people that stop "doing" also often forget how to "think" about the actual problems properly.

I do this job and I'd say it's only enjoyable for now because I have complete freedom and I'm still learning. Splitting up the parts: Design architecture Wire up pipelines (once architecture is decided, this tends to declarative, connected is choosing schemas). Data science

I'd definitely not want to be stuck doing stage 2 forever, would prefer 3. I think you're saying that you enjoyed a job which was some 1 and some 2. I'm sure there's someone out there who wants to wire up pipelines with no engineering and no analysis all day but I'd imagine it's a rare breed.

Edit: I think the important distinction is team/company size. Doing a bit of everything as a 1 man team is challenging, if you have a team where devops/engineering/reports/tools have been chosen/built/standardized by specialist and you really are just wiring pipelines up, I think that would be tough. On the other hand being in a small team condemns you to always be doing the same fractions of work because there's noone to hand off to.

> I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated...

I think this is the key idea there. It's good that you found a situation where you're both appreciated, and compensated for it. It's too easy for engineers to be devalued as replaceable cogs in the pipeline of things that need to happen to bring in revenue.

Yeah, seems like an aging configuration among quants and various teams in IT at many finance companies.

Finance may have been ahead with adopting data science but it seems that many others reached parity or even surpassed in capabilities.

I'm working for a small hedge fund in basically the same role, do you have any resources you could recommend for learning how to do these things the "right" way?

Reply on news.ycombinator.com