remilouf

In this case no one technically competent was here to manage them. There were no expectations.

I love SQL. But it hard to get other DS on board who think that 40 lines of Spark is better than a 10 line SQL query.

The only thing that worries me with SQL is when having to write UDFs for, say, computing a Z-score. But maybe it's just because I have never done it? Do you have any good resources about this?

Thank you for your answer, it was eye-opening. The story is that they opened that department a year ago and needed buy-in quickly so needed to jump-start it by hiring consultants. Whether hiring consultants was a good idea or not, I understand how things ended up where there are. A mix of inexperience and huge pressure to deliver.

Now that we have buy in the pressure isn't so bad. I have a startup background and this is one of the reasons why I was hired. I am not too worried about having to convince my manager, she's great and is the one who started criticizing the legacy (she arrived a few months ago) and asked me to dive in and give my opinion.

She agreed to let me work 30% of my time on this, and the rest on delivering direct value to the clients. The related tickets will be part of the sprint. I will do as you say now, and create items that address the concerns.

Thank you for taking the time to answer thoroughly !

> If it is only used by data science team then you can rip apart the solution and work towards a more logical infrastructure where all you are doing is cleansing, normalizing and deriving features and these become your central feature repository which your team can pull and build models.

It is only used by data scientists. What do you mean by a feature repository? How would you organize it so people can push new features? This sounds very interesting.

> you are talking about denormalization as something you are planning to do, that should have been the starting point of any HDFS/SPARK/Parquet based solution.

It is something that we have to do, but the table have been dumped as is in S3 and every project rebuilds the whole derived dataset regularly. Since these operations are very brittle (a lot of manual work and even transformations performed in notebooks), this is something people dread doing. I am trying to secure this at the moment by writing Makefiles that remove human intervention, but at the end of the day I would like to avoid people spend hours waiting for new data when they need it.

> I can suggest tools for explorations, data quality check etc.

I would appreciate it. Put simply, we get data about the evolution of the stock of clients, transactions with their clients, product descriptions, etc. that is dumped into S3 (I scheduled a chat with people upstream to see what happens). We have 3,4 projects for each client. What currently happens is every team writes the same code to build features in their separate repositories, this code is re-executed every time new data arrives (weekly). These features are then used in prediction models.

Besides the brittleness of the process, I found that people are reluctant to analyse the data because it takes an unreasonable amount of time.

Well, when several people are working on the same project they "share" the transformed data by connecting to the same EC2 instance. The way data is transformed is via 4 scripts, 2 notebooks and a bunch of manual operations, so no one really wants to touch that. I spent my 4th day working with a contractor to write a Makefile that reproduces all the steps for ONE project.

I talk about adding infrastructure in my original post, but I'm very well aware that my time is currently better spent consolidating the existing as much as I can so the clients can get correct results faster.

I did use BigQuery in the startup I was working for before, and it worked wonders for our 12Tb of data. I think it would be a bit overkill in our situation---even though not having to manage a DB is great.

For some use cases I would totally agree :)

I second that. I find that it helps junior dev find the sweet spot in commit granularity.

I feel it reduces cognitive load, like proper formatting of code.

Do you enforce consistent code formatting in your codebase (using CI, maybe)? If so, why wouldn’t you do that with commits?

I don’t completely agree. By taking the time to write commit messages you are documenting your code at a fine-grained level. Think of it like a short email that explains to your (future) colleagues and future self what you changed and especially why you did it.

A good git log prevents unnecessary headaches and saves a lot of time. Typical situation: you just found a couple of lines whose purpose is not completely obvious. If the author is still there, you both waste time discussing. If she/he left the company and you have to figure it out by yourself. Now imagine using git log to identify when these lines where introduced/changed and getting an explanation as to why.

So yeah, writing good commit messages can feel like a waste of time. Like writing good code can feel like a waste of time. You should not be doing it to please your colleague, but rather to contribute to a maintainable codebase. It’s worth the extra minutes here and there.

Bonus point: less documentation to write.

This. I was in an extreme situation like you were (boss abusive with everyone in the company, constant threats and sometimes physical abuse).

- First, see if you can find other employees in the same situation. Being a group helped us. - Document everything. Commit on git first thing in the morning, and before leaving. Write down every conversation you had right after you had them—we tend to forget very quickly. Dump your calendar and emails regularly. You can find scripts to dump Slack too. If your boss has written anything abusive them, it is also a good idea to take screenshots. - I guess moving is not an option for your family? Otherwise start looking in other markets. - Freelancing is a good option otherwise. I know it sucks to do this on top of your job, but you only to save enough to get yourself out of this.

Once you’re out, and only when you’re out, find a good lawyer. Do not hesitate to sue. It’s not greed, it’s reparation for the abuse. You deserve it.

Good luck to you.

My trick is very simple and very low-tech. Take a piece of paper and a pen and draw a diagram that sums up the big picture of what you are trying to achieve (architecture of the software you’re writing for instance). Take as long as you need to, and when things start to fall naturally into place and you are eager to start, pick one part of the drawing and start working. Avoid distraction at all cost (music does help, especially in an open space).

I developed this when I was in academia and it transposed very well (better?) in the software industry. If you do that regularly it also helps course-correct when needed.

By being explicit about the assumptions they make. It does not mean they are not biaised, but that you at least know what poison you chose.

And I believe you. In the end, no language is better than another one in absolute. It all depends on the use case. In mine, Elixir and Go happen to outperform everything else I know but they are probably terrible somewhere else.

For a MVP you’re right. But a few years (months) down the line, you may find that some alternatives scale better. Elixir, for instance.

As someone who also develops in Python and Nim, I respectfully disagree. There are a lot of use cases where I would pick them over Go, but the important term here is “production grade”.

Sure, you can bootstrap something really quickly in Python and Nim, but when it comes to creating beefy APIs, I find that Go offers a much shorter development cycle. Sure it is slightly more verbose, but when your code compiles it usually works (which is partly due to the way Go deals with errors). If you need to handle a large number of requests, scaling with Go is a breeze and a nightmare in Python.

I don’t know how this works in the US but what comes to mind is voting rights. It may seem unimportant now, but might come and bite you in the a later. It really depends on your relationship with this person.

Maybe not on the “life is short” metric. It is insanely fast to write production grade code in Go :)

“This is water” the transcript of a commencement speech that David Foster Wallace gave. Probably the best thing he ever wrote (and shortest :) ).

Erlang and Go.

I have a background in data science, so have a lot of experience with Python. I’d always wanted to do more backend stuff: push my models in productions, do the ETL work myself... But quickly got frustrated with Python.

Go and Erlang have solved a lot of my pain points. Erlang is a bit of a strange beast for people who come from procedural languages, but it allowed me to build ETL pipelines that are more robust than anything I could do with Python in less than half the number of lines. Even for heavy computations, I now find myself spinning Python nodes to do the heavy lifting with Pytorch and co, and doing all the orchestration with Erlang. It also made me a way better programmer.

I use Go for computation-heavy APIs that don’t require a lot of data massaging. It’s a very efficient, weirdly comforting (!) language.

Now I think I got the bug, I really want to learn Clojure and Nim this year :)

Before even considering it I would ask the CEO why she’s leaving and the employees. Some people have very good reasons to leave a sinking ship (family being the most common), but others leave because of systemic problems in the company. I’ve seen CEOs burning out their team by creating a climate of fear before leaving when things go sour, claiming everyone is « incompetent ». I really advise you to spend A LOT of time talking to employees.

Cons:

- You’ll have to learn about a lot of different people at the same time. In early stages startup you get to meet people one after another and earn their trust one after another. This is different.

- You probably have a different vision than that of the previous CEO, and you’ll have to go the extra mile to align everyone.

- You inherit a culture you didn’t shape. You’ll probably have to change that, and it will take time for people to adjust. Just lead by example. - All of this takes time, and meanwhile the company needs to survive :)

Pros:

- You’ll ramp up your people skills like never before. Gaining the respect of an entire team as a CEO, and (sometimes) repairing damage will take a lot of courage and empathy.

- Obviously, if you turn this around you’ll get major valley cred. And, never underestimate this, a huge satisfaction.

If you decide to go that way, good luck, and you already have my respect!

I’ve struggled for a long time with paper scraps to keep track of what I’ve been doing and read about the following alias to add in your .bashrc

`alias did="vim +'normal Go' +'r!date' ~/did.txt"`

Then type `did` in your terminal and write what you just did.

More generally, you can get a great boost of productivity by mastering your text editor. I have a preference for Vim, but have nothing against emacs.

+1 for i3. Tiling window managers dramatically increased my productivity. No more “where did that window go?” headache.

My wife and I went through something similar a little less than a year ago (baby die at 6 months from a heart failure). Every now and then I still get the occasional diaper ad popping up.

What disturbs me the most are colleagues who said it was my fault since I hadn’t shared my grief on social media, and algorithms could not learn (I work in data « science »). As if targeted advertising was a law of Nature and there was nothing we could do about it. Just deal with it. And blame yourself for not showcasing your pain in front of hundreds of people.

The issue with algorithms is not that these events are statistically unlikely (unfortunately). They just have memory, enough memory to be accurate. So the day happy messages are replaced with the word « death », it hasn’t had the time to learn baby was not there anymore. There may be ways to improve that, but I bet you’d be trading accuracy for the remaining 95% people. Guess what version wins at a business meeting?

I’m not blaming colleagues who write these algorithms. They didn’t know this could happen, they didn’t think about it before this happened to me. We’re all math and CS geeks priding ourselves in [insert your metric here] score above the 90% mark, and that’s all we optimize for. I’ve heard a joke once that the difference between science majors and humanities major was that scientists build the atomic bomb, humanities major explain to them why it’s not a good idea to use it. Targeted advertising is not as bad as the atomic bomb, of course, but you get the idea.