Ask HN: I have 10 yrs of Exp. Failed 4 takehome projects. What am I doing wrong? | Hacker News Reader

Ask HN: I have 10 yrs of Exp. Failed 4 takehome projects. What am I doing wrong?

25 pointsformulathree85 comments

I'm actually a super senior/staff engineer with roughly a decade of experience. This current interview run has been demoralizing because I literally put a lot of effort into 4 different take home assignments and all 4 came back as failures with zero feedback given. I'm at a complete loss for what I'm doing wrong. I initially went in with a lot of confidence but I'm getting totally shut down on every submission.

Here's an example of a recent one:

https://github.com/anonanonme/takehome-sample

takehome instructions: https://github.com/anonanonme/takehome-sample/blob/master/README.md

takehome readme: https://github.com/anonanonme/takehome-sample/blob/master/documentation.md

Can anyone provide some legit criticism? Or is the person evaluating this just being unreasonable?

View on HN

A take home test as a first round filter usually means they are not that serious about hiring and it costs them no time, money. Meanwhile you need to invest 4 hours (they actually want you to spend 8 hours) for a position you have no idea if you are really interested in yet.

The chance you will get ghosted? Very high.

The four hours could be better spent.. you are more likely to get a response by cold emailing other companies.

Send them open source work and ask them to judge your ability/style based on that. If they are not interested they were never going to judge your submission based om your style / abilities and the four (8) hours can be better spent.

Good tip. I think I'm never going to do a take home assignment again.

This is where external recruitment agents can be a good thing. You get to talk to an intermediary (a biased self-serving one for sure) and ask them about the interview process etc. They normally have more than one thing to send you to, so happy to discount X to send you to Y which might have a better process.

When using them I find you need to keep your wits about you. Some are sharks. Always confirm whether or not they are putting you forward for something being the main thing. I have been put forward for roles without knowing then got unstuck because 2 agents put me forward for the same role.

I don't know, I've been ghosted by 0% of take home tests. I usually go for them because the odds of getting in are better than resume spamming, and usually I'll be grinding leetcode anyway.

There was even one company that was clearly not a culture fit; I think the HR was a little racist, based on the tone in the email. But I passed the filter test and they probably had to interview anyway.

First round filter tests are usually along the lines of Fizzbuzz though. More than 20 mins and it's a red flag.

I've never been ghosted by a take-home test and have gotten many offers >$500k from interviews that involved take-home tests.

This is my anecdote, you can take it for what it's worth.

The key distinction is “take-homes as the first round”. IME OP is right, do not do a take-home as a first interview round before you even have any chance to ask them questions. They’re not serious and will likely ghost you.

Even if they are serious, and you ask questions after the take-home then find out they don’t believe in version control and testing, you still wasted your time!

I've done take homes as the first round.

I've never failed a take-home round and I've never been ghosted.

Again, this is my personal experience.

I would assert that the mistake is accepting a take home assignment in the first place. I would walk away at the mere suggestion, because they do not respect the value of your time.

If they do not value your time while you are interviewing, there is zero reason to believe they will respect it once you are hired.

Good call. I think I'll be following this advice for the future.

There's a million reasons why you could and will get rejected that have nothing to do with the code you've written for the take home exercise. My usual solution is to ignore the losses and carry on with the grind. You only need to win once. Good luck!

Thanks for the tip. Yeah I know this.

Just wondering though with 4 failures in a row, I think there is a good chance something is wrong with my code so I wanted to make sure with this post.

I've been doing the grind for years, but this is the first time I had so many take home projects.

Additionally, I never gotten hired by doing a good take-home either. Which is weird.

Additional question: What's your guys take on take-homes: Anybody ever successfully get hired by doing well on a take-home?

I think you’re unfortunately in a sellers market for jobs. It was already kind of a terrible experience, but given all the recent layoffs, you’re going up against lots of competitors, and the hiring companies don’t have a lot of incentive to provide feedback.

As ridiculous as it may sound, don’t take it personally.

>As ridiculous as it may sound, don’t take it personally.

Yeah thanks for the tip. I'm not taking it personally. But I still do consider this practice rude and inappropriate. It's just the rudeness is not directed at me personally but at everyone in general.

It still will reflect on the company regardless of whether anyone takes it personally.

I’ve hired a lot of people and tbh it’s so much work on top of your existing work, with a scramble every time to find slots that fit for several people, the tediousness of doing the same interview over and over, it’s just hard to find time or energy to provide feedback, you’re mostly just focused on moving forward the candidates you want to.

I know it’s shitty but that’s the reality, it’s nothing personal. When doing the interviews I always give my best to be engaged and respect their time, but other things tend to fall through the cracks

No. If the candidate spends four hours on a take-home it is basically a moral obligation that you give feedback.

The exchange is literally unbalanced here. You ask for four hours from the candidate. The candidate asks for 1 hour from you for feedback.

The candidate is also doing things on top of existing work to spend time and energy on a take-home.

There's no two ways about it. Morally it's the wrong thing to do.

Yeah ok fine sue me. We’re all understaffed and overworked.

Honestly i dont think the code was the problem, maybe hr just didnt like you

This is possible. But I did do an interview with HR screening and they passed me which gave me access to this takehome. I know I seem a little abrasive in my replies here, but I'm being genuine... a lot of the criticism I'm seeing doesn't make sense or seem overly pedantic to me.

But! there was one huge flaw with my code! like huge. Only one person in this entire thread caught it. This is a rejection level flaw for sure. It may have been it but this thread is making me think it might not be.

What's astonishing is that nobody caught it. Nobody but one guy. This flaw would've gone past all unit tests all type checking and all integration tests. No engineering process would've caught it other then analyzing the code line by line and seeing exactly what each step does.

So it makes me think that if more or less no one on this thread caught it... it's likely that the reviewer didn't catch it either. I'm thinking it's more likely some superficial aspect of the code screwed me over even though this flaw is huge. Like no unit tests or something.

It's not even some obscure flaw either it's very visible. the ZRANK call isn't doing anything. it's an empty command with discarded output... a snippet generated by code from chatgpt which lied to me about what that command does. I tested that code extensively and it worked fine.

This mistake deserves rejection, but like I said, I somehow think that this wasn't what got me rejected.

Why do you think random people on hackernews are going to take the effort to read your code ?

One easy change: You could take a bit more care to make your code tidier. As an example, you're wildly mixing " and ' without any good reason (copy paste from somewhere?). The style of your line breaks also varies a lot. Just use any formatting tool and your code will look much more cohesive. Some structure in the packaging would also go a long way (folders don't cost anything).

This might seem superficial but as a super senior staff, I'd expect that you mentor your junior coworkers in how to build large yet maintainable projects. One important aspect is readability and consistent style. That's why black becomes more and more used even though some of its rules are fugly.

Second not so easy change: I find "The code was also not fully organized for testing" - this actually hints at you either not writing a lot of tests in your day to day life or being very inefficient in producing tested code. You also could have at least included one sample test.

The wildly mixing is because sometimes I want to use quotes within quotes: ' "a string" '.

I default to this " ". But I switch around when needed. This is standard python methodology. Anyway..

This kind of thing doesn't matter. It's pure aesthetic, it's ocd stuff in my opinion. Whether you use " or ' has zero consequence to readability, maintainability, safety, and performance. Literally zero. Your code can grow to a billion lines this still doesn't matter. This is actually an ocd instinct that you should try to separate from your rational thinking. The inconsistent quotes * feel * wrong, but rationally they have zero consequence on any metric. It's just a feeling not an actuality.

Less experienced programmers often get caught up in stylistic issues but fail to realize when style is actually of zero rational consequence. Not all styles have zero consequence but some of it does.

>The code was also not fully organized for testing

Only literally 2 functions are unit testable here.

One of them is so basic that it's not even worth testing.

All unit testable functions are segregated away from IO so from a fundamental standpoint it's pretty much unit testable to the full extent.

The reason why I wrote that it's not is because the random path generation function has a bit of a flaw. It uses random number generation, so it can't be unit tested. I can't predict the output in the assert because it's random.

That's it. The amount of logic amenable to testing is so trivial and little it's not worth it here. I could segregate the random number out of the path generation function deterministic and testable but I made a judgement call here. It's a take-home and segregating the code like that leads to an API for the function call that's less intuitive. The function would need random seeds as input parameters.

Ideally, yes I shoulda done that. But this is a valid shortcut. I would say though, the majority of people aren't able to even realize how random number generation effects unit testability.

The unit testing thing is valid. Everyone mentioned that even though what's testable is so little. I'll just have to cater to the current dogma of the programming world.

One thing to bear in mind is that people reviewing applicants are looking for a reason to reject applicants.

It used to be different when it was hard to find people. We would be looking for excuses to accept people - e.g. the code is a bit rough, but maybe they didn’t put too much effort into the take home - let’s get them in for an interview just in case…

Now with so many applicants it will be much more a case of who can we reject so we don’t spend all our time interviewing…

Also, don’t forget, the people reviewing your code may not be the senior engineers, bend with the wind, run your code through a standard formatted so that it will match expectations. Do the unit tests even if you think it’s pointless.

Play the game.

Solid tip. Will follow. Thanks.

Judging someone because they mixed double and single quotes is silly. That is easily fixable with a linter that they might have decided not to include for the sake of time.

This kind of comments are kind of useless it should be self explanatory

" # returns the path and current count that was just made.

return {"path": path, "count": int(data[0])}"

also maybe extract variables and name them better like:

"pipeline_execution_result = await pipeline.execute()

count = int(pipeline_execution_result[0])"

same here:

    return [{"path": path.decode('utf-8'), "count": count} for path, count in response]

what's in settings.py should probably be in an env file

overall: variables and methods name could be clearer, use good names instead of bad comments, extract variables, don't define methods inside of methods

it's definitely not BAD code, but I'd expect a guy with 10YoE to do better

That’s pure nagging and notmystyleism, except for the env bit.

>what's in settings.py should probably be in an env file

Have you used Django? Django follows the pattern of a settings file. No environment variables.

A global env file imo is definitively worse. It forces you to write code outside of the python ecosystem to extract these variables. Additionally python code references an environment variable that's not explicitly set by the env code could hit some logic errors or unexpected state.

What I would do if I had more time is have the settings file reference an env. The main benefit is that settings can be reused across apps and env errors are localized to a single point of failure instead of being littered throughout the app as references to env vars.

>This kind of comments are kind of useless it should be self explanatory

They are useless. I agree. I put the comments there in case someone disagrees. Doesn't hurt in my opinion.

>also maybe extract variables and name them better like

You mean with patten matching. No that's actually syntactically less appropriate here. The return value was not a structured product type. It was not a tuple.

The return value was a list which implies variable length. Should the list change in size that would change the pattern match. This is most likely just bad typing on the library. But my handling of said type is appropriate. Also it's super minor.

>variables and methods name could be clearer, use good names instead of bad comments, extract variables, don't define methods inside of methods

I'm actually with you on this one. I prefer clear names over elegant names. But this is not overall sentiment among the majority.

Overall people prefer an elegant name over a very descriptive one simply put of some universal intrinsic ocd instinct they all have... even though short elegant names could provide zero informational value.

I'm just catering to the majority here. Make myself a comment Nazi as most people don't disparage that and use elegant names as most people prefer that over longer descriptive names.

>it's definitely not BAD code, but I'd expect a guy with 10YoE to do better

Have you seen code written by people with 10yoe? What you will find is overall 10 yoe doesn't converge on your personal view perfect code.

It converges on their view which likely is wildly different from your view.

But this thread has been extremely informative on that fact seeing how literally everyone's view is completely different and how everyone thinks they're own personal view of the universe is the enlightened path.

The common theme: is lack of unit tests. But to further illustrate the diversity of opinions... You didn't even touch on testing. It took a back seat to "bad comments".

It is common to load the variables inside the django setting files from an env files. This way in different environments you can have different values, and the secrets are not committed to the repository, but managed externally.

Yes I mentioned this.

orgs are chaos, “person” “reason” are both assumptions. for example perhaps there were 8 interviews that week for one position and it was earmarked for an internal candidate anyway. perhaps the position went away. perhaps your submission was too good and they hired a junior instead. perhaps the recruiter ran the standard process and the hiring manager didn’t even see your submission. perhaps your submission was best but the reviewer didn’t understand it. perhaps the reviewer was 24 years old. perhaps they hired someone who had higher status companies on their resume and that person didn’t even have to do the test. perhaps the manager is on a pet functional programming kick this year. perhaps you like FP but the manager had a bad experience with that this year. perhaps they wanted a woman. perhaps you’re a woman and they wanted a man. perhaps the manager is a red bull code smash bro. i could go on all day

Could be, but with 4 in a row? I'm inclined to think something is wrong with my code, that's why I put an example up.

People are biased, and I'm biased as well, so I hope other people can see past it and see what's wrong with my code.

The code overall looks reasonable and I don’t write much python but:

- requirements.txt is the standard for specifying deps, right?

- making a README explaining the project might be good (I see you have this, but maybe switch documentation.md to README.md)

- tests?

- building the flask application via a function is a bit more testable

- reading settings from ENV is preferred (12 factor apps)

- output not sanitized for /test endpoint

- generator in util could have been pulled out probably? It’s created every time

- generate test path function seems a bit longer than it should be — all those lambdas should probably just be functions since you’re going to use them a lot

- did you have to define your own Json type? Is that complete? Where’s null?

- url generate function should probably template hostname — that’s more important than host, most of the time for running in different environments

- trailing slashes matter in flask, evidently, and every request without one gets redirected (test suite would have caught this)

- on the usage of redis, I wonder if scanning + in-memory aggregation is better… zincr/zrank/zrevrange is good, but you’ll have to hold all data in memory (and receive it in one large response) and logN anyway, might as well do it simply with a set with a dynamic prefix and scan while building the output data structure as you go.

- do your API endpoints return JSON?

- error handling around points of failure like redis

— your app goes down if the connection is flaky right?

- zrevrange is deprecated now btw

Hard to tell for far they wanted you to go, but asking might have made sense… some nice-to-haves:

- healthz endpoints?

- metrics?

- tracing?

- error reporting (ex. Sentry)

Huge thanks for the feedback!

>- requirements.txt is the standard for specifying deps, right?

poetry is a new thing. I just tried it for this project it replaces requirements.txt with pyproject.toml

>- making a README explaining the project might be good

I did. It's documentation.md. I didn't title it README.md because I wanted the front page on github.com to show the takehome instructions rather then my docs on it.

>- tests?

This was addressed in the documentation.md. I was given 4 hours to work on this problem so I just used manual tests.

>- building the flask application via a function is a bit more testable

Flask is an IO app. It's inherently not testable via unit tests because it's a server. You'd have to build integration tests around an entire server which is huge overkill for this project. Testable logic is usually pure and stateless, that's located in utils.py. It's ok this is good criticism.

You can monkey patch or make your code 10x more complicated with modules that accept mockIO for dependency injection but I'm actually against this style of programming as it over complicates the code with little benefit.Trying to test Flask by mocking everything out is basically just testing the mocks. Better to do this haskell style and segregate IO from pure functions (see utils.py). IO can't be unit tested.

Not many people understand the "testing" philosophy I'm following here so this is good feedback. I should sometimes just follow what's popular but ultimately not the "best" way in my opinion.

>- - output not sanitized for /test endpoint

this I don't get? Sanitized? The http request returns typical python structs, which are automatically converted to Json by flask/quart.

>- did you have to define your own Json type? Is that complete? Where’s null?

Nope don't have to, type checking is optional in python, but why not? You have to do this to get correct type annotations to everything. You're right about this missing Null/None. It all still type checks.

>- generator in util could have been pulled out probably? It’s created every time

Why pull it out? A generator is cheaper then creating a list from a list comprehension every time. instead of storing every value in a list and iterating through it, I just iterate through a generator. Saves memory and runtime cost is still O(N) regardless.

>- - url generate function should probably template hostname — that’s more important than host, most of the time for running in different environments

Sure but the context is a takehome project and it's running in docker-compose as specified by the directions. I mean yes, I can make that utility function more general for sure.

>- trailing slashes matter in flask, evidently, and every request without one gets redirected (test suite would have caught this)

I'm aware of this, it's not a mistake. I left it in as valid. The redirect does not get counted so api/xxx/ is equivalent to api/xxx

Your way is NlogN sorting and constant time inserts. The current way is zero cost sorting and logN inserts.

LogN is blazing fast, while nlogn can get slow if N gets too large. See this for relative visualization: https://i.stack.imgur.com/osGBT.jpg. Given the picture I would say my way is better. .

>- - do your API endpoints return JSON?

Yes this is the default. If you return anything that's equivalent to that JSON type I defined in utils.py, flask will automatically return serialized json.

> - error handling around points of failure like redis

Flask/quart runs under the hypercorn server (see the dockerfile). Additionally If a handler throws an exception the user automatically gets a 500 error from flask it's handled exactly as you would expect an http server should handle it. The server does not crash.

>— your app goes down if the connection is flaky right?

No it does not. hypercorn remains running if the python worker crashes. But of course if hypercorn itself goes down then it's done. This of course can be mitigated by systemd but that's overkill for this project.

>- zrevrange is deprecated now btw

Yeah your right. My mistake. Still works though. So it's marked for deprecation, but not actually deprecated yet.

>-healthz endpoints?

>-metrics?

>-tracing?

>-error reporting (ex. Sentry)

For a four hour take-home project? These things weren't even in the spec and how these things and if these things are implemented are extremely variable per company across the industry.

You know overall your post has been enlightening but not in the way you think because we are clashing on most of these points. I can see how developers can literally disagree on everything and how code reviewers can make a ton of assumptions and have a ton of arbitrary opinions. Not to mention varying levels of experience and what experience for that matter causes them to make completely different judgement calls. Don't take this as a slight, I'm sure if I was in your position I would be reviewing your code in the same way and you'd see me in the same way as well.

I'm starting to think take home assignments are just bad. Even worse for evaluating programmers then algorithm interviews because there's so many biased variables here that programmers are just clashing on. Maybe it's a good filter for finding programmers who view the programming world in the exact same way they see it.

> poetry is a new thing. I just tried it for this project it replaces requirements.txt with pyproject.toml

Ah thanks, I saw this later -- the pyproject.toml was very illustrative.

> I did. It's documentation.md. I didn't title it README.md because I wanted the front page on github.com to show the takehome instructions rather then my docs on it.

Is there a reason for this? I would think this was a good chance to show off your ability to write succinct/standardized documentation w/ sections that anyone else would be glad to read/see.

> This was addressed in the documentation.md. I was given 4 hours to work on this problem so I just used manual tests.

Yeah I can see this, but I think "automated tests" are quite big in peoples' minds. Feels like it would be a checkbox on the item.

Budgeting some of the time to make 100% sure your code was tested seems like it might have been a good idea.

> Flask is an IO app. It's inherently not testable via unit tests because it's a server. You'd have to build integration tests around an entire server which is huge overkill for this project. Testable logic is usually pure and stateless, that's located in utils.py. It's ok this is good criticism. > > You can monkey patch or make your code 10x more complicated with modules that accept mockIO for dependency injection but I'm actually against this style of programming as it over complicates the code with little benefit.Trying to test Flask by mocking everything out is basically just testing the mocks. Better to do this haskell style and segregate IO from pure functions (see utils.py). IO can't be unit tested. > > Not many people understand the "testing" philosophy I'm following here so this is good feedback. I should sometimes just follow what's popular but ultimately not the "best" way in my opinion.

I disagree here -- table stakes for a good API is integration and E2E testing -- and they're the easiest to do (no need to manipulate a browser window, etc).

I agree with you on not testing the mocks, but for that I personally would bring in a library for this like testcontainers for python: https://testcontainers-python.readthedocs.io/en/latest/READM...

IMO E2E tests are the most valuable tests.

> this I don't get? Sanitized? The http request returns typical python structs, which are automatically converted to Json by flask/quart.

Sorry INPUT is what I meant to write there -- what happens if you get a -1 ?

> Nope don't have to, type checking is optional in python, but why not? You have to do this to get correct type annotations to everything. You're right about this missing Null/None. It all still type checks.

Right, but this is what I mean -- when I read that I wondered if it was complete, and it wasn't. That's an unnecessary ding to take.

If you're going to type it, it should type correctly, or you should use pull in the typings from somewhere else, IMO.

> Why pull it out? A generator is cheaper then creating a list from a list comprehension every time. instead of storing every value in a list and iterating through it, I just iterate through a generator. Saves memory and runtime cost is still O(N) regardless.

This is going to build objects dynamically every execution right? That's what I thought could be avoided. Less about lists vs generators, more about doing things that look like they allocate so much.

Do you think there's something you could do to create less garbage during the function execution? If not then you can disregard that point!

> Sure but the context is a takehome project and it's running in docker-compose as specified by the directions. I mean yes, I can make that utility function more general for sure.

It's not about making it general per say -- it's more like what is your instinct at this point.

My instinct personally (and granted I spent like... 30min to an hour just reading all the code) is that I always take my host/port from ENV, and everything else possible. It's just something you know is going to come up.

> I'm aware of this, it's not a mistake. I left it in as valid. The redirect does not get counted so api/xxx/ is equivalent to api/xxx

Ah OK, but this isn't what they asked for, right? If we want to be really pedantic, your endpoint returns a 3xx for every single error.

Redirects are not followed automatically, it depends on the HTTP client, right? So one that doesn't follow redirects would fail this test. It's unlikely, but maybe this code failed automated testing for a little reason like that.

> Your way is NlogN sorting and constant time inserts. The current way is zero cost sorting and logN inserts. > > LogN is blazing fast, while nlogn can get slow if N gets too large. See this for relative visualization: https://i.stack.imgur.com/osGBT.jpg. Given the picture I would say my way is better. .

The measurement endpoint will be called far more (100x, 1000x, ...) than the stats endpoint -- it's the hot path.

Also, note that sorting is zero cost, but retrieving the data will still be in N time including network round trips, which will likely dwarf any time you spend sorting in local memory. Even the conversion to JSON will dwarf the time you spend sorting for large N.

You're right about the complexity -- but note that ZREVRANGE is O(log(N)+M), and in this case we KNOW that M = N, so IIRC that's O(N).

Regardless of the complexity, I personally wouldn't make that tradeoff until it was necessary -- speed of the hot path is more important in my mind, and there are other ways to manage speed of the stats endpoint.

Also, the point of memory usage being potentially large is still relevant. My instinct is that when I see something returning a possibly very large piece of data, setting up for streaming is the way to go. Doing a scan you have the possibility to use far less memory for an individual request, and you could actually stream responses back in extreme cases, so you can avoid holding the entire result set in memory at all.

If you want to get really fancy, you can get creative with how data is stored on the redis side and get the size to N.

> Flask/quart runs under the hypercorn server (see the dockerfile). Additionally If a handler throws an exception the user automatically gets a 500 error from flask it's handled exactly as you would expect an http server should handle it. The server does not crash.

Ahh OK, I did see hypercorn in the Dockerfile and forgot about it, I see how that your worker would be restarted by hypercorn.

The point about error handling still stands with respect to reporting though -- you generally want to see error handling in code, rather than just leaving it up to the runtime to fail the task and carry on.

Adding this would have prompted you to think about some sort of error reporting, which is generally good practice.

> For a four hour take-home project? These things weren't even in the spec and how these things and if these things are implemented are extremely variable per company across the industry.

The things you include/don't include by instinct I think is valuable signal.

Like I said, they're nice-to-haves, but error reporting is probably the big one there. In production errors should to be gathered somewhere.

You can make the argument that they'll be collected by a logging system and that's why you've left them out, but I also don't see any structured logging.

> You know overall your post has been enlightening but not in the way you think because we are clashing on most of these points. I can see how developers can literally disagree on everything and how code reviewers can make a ton of assumptions and have a ton of arbitrary opinions. Not to mention varying levels of experience and what experience for that matter causes them to make completely different judgement calls. Don't take this as a slight, I'm sure if I was in your position I would be reviewing your code in the same way and you'd see me in the same way as well. > > I'm starting to think take home assignments are just bad. Even worse for evaluating programmers then algorithm interviews because there's so many biased variables here that programmers are just clashing on. Maybe it's a good filter for finding programmers who view the programming world in the exact same way they see it.

Yeah, that's a pretty common take (see all the comments in this thread).

I'd disagree that it's worse than algo interviews, because algos are rarely the most important thing in a professional code base, especially one with that's about web services and not something more demanding.

All this said though, the only way to know why they passed is to ask! Unfortunate that you can't get any feedback.

>Is there a reason for this? I would think this was a good chance to show off your ability to write succinct/standardized documentation w/ sections that anyone else would be glad to read/see.

I threw the docs in the README for the submission, but for HN readers they want to see the problem first. The submission didn't even include the specs.

>Yeah I can see this, but I think "automated tests" are quite big in peoples' minds. Feels like it would be a checkbox on the item.

Yeah you're completely right here. I'm gonna go with it next time.

>I disagree here -- table stakes for a good API is integration and E2E testing -- and they're the easiest to do (no need to manipulate a browser window, etc).

Yeah but many companies don't do integration testing as the infra on this is huge and writing tests is more complicated. It's certainly overkill for take home. I would say you're wrong here. Completely. This company was not expecting integration tests as part of the project at all.

I've worked for start ups most of my career, and I'm applying to start ups. It's mostly the huge companies that have the resources to spend the effort to have full coverage on that.

Integration tests are Not easy, btw. Infrastructure is not easy to emulate completely, how would I emulate google big query or say aws iot on my local machine? Can't, most likely integration tests involve actual infra, combined with meta code that manipulates docker containers.

>Sorry INPUT is what I meant to write there -- what happens if you get a -1 ?

You get a 500. Which is not bad. But you're right a 400 is better here.

>Right, but this is what I mean -- when I read that I wondered if it was complete, and it wasn't. That's an unnecessary ding to take.

I feel this is such a minor thing. I'm sure most people would agree if you dinged that you'd be the one going overboard, not the candidate.

>This is going to build objects dynamically every execution right? That's what I thought could be avoided. Less about lists vs generators, more about doing things that look like they allocate so much.

You're going to build objects dynamically every execution Anyway. This saves you the extra intermediary step of not having to save it to an extra list.

    numbers = [i*2 for i in range(500)] #allocated list in memory: 500*sizeof(int) (big) 
    for i in numbers:
       print(i + 1)

    numbers = (i*2 for i in range(500)) #allocated generator in memory: sizeof(*generator) (small)  
    for i in numbers:
       print(i + 1)

Look at the above. Same concept. Generators are drop in replacements for actual state. State and functions are the isomorphic (ie the same thing). Building a generator is cheaper then building a list. You should read SICP about this topic of how functions and data are the same.

>Ah OK, but this isn't what they asked for, right? If we want to be really pedantic, your endpoint returns a 3xx for every single error.

It's a reasonable assumption that all users assume /xx/xx equals /xx/xx/. It's also reasonable for a client to handle a 302 redirect. The specs didn't specify the exact definition on either of these things.

>The measurement endpoint will be called far more (100x, 1000x, ...) than the stats endpoint -- it's the hot path.

So? logN is super fast. You have 100 objects binary or whatever indexed insert redis uses gets there in at most 4 or 5 jumps. So even if it's the hot path redis can take it.

NLogN is slow enough that if N is large there is noticeable slow downs EVEN when it's a single call. Recall that redis is single threaded sync. NlogN can block it completely.

Either way this is all opinion here right? Ideally the results are returned unsorted. the client is the best place to sort this, but that's not the requirement.

>Also, note that sorting is zero cost, but retrieving the data will still be in N time including network round trips, which will likely dwarf any time you spend sorting in local memory. Even the conversion to JSON will dwarf the time you spend sorting for large N.

Of course the act of reading and writing data will cost N regardless. But your reasoning is incorrect. Look at that picture again: https://i.stack.imgur.com/osGBT.jpg. O(NlogN) will dwarf O(N) when N is large enough. It will even dwarf IO round trip time and has the potential of blocking redis completely on processing the sort.

>Also, the point of memory usage being potentially large is still relevant. My instinct is that when I see something returning a possibly very large piece of data, setting up for streaming is the way to go. Doing a scan you have the possibility to use far less memory for an individual request, and you could actually stream responses back in extreme cases, so you can avoid holding the entire result set in memory at all.

Sure but look at the specs. It's asking for http requests. It's not a streaming protocol here. I mean at this point if you were evaluating my code for a job you'd be going to far already as your going off the rails completely.

>The point about error handling still stands with respect to reporting though -- you generally want to see error handling in code, rather than just leaving it up to the runtime to fail the task and carry on.

I mean I could build a full logger and error handler framework here. I'd put custom handling in a decorator and have the error handling invisible in the http route handlers. Flask already does this, but it just returns 500 on all exceptions. That is the most elegant way imo. But again this is a bit outside of the scope of a takehome, given the framework handles a lot it and the extra effort required.

>If you want to get really fancy, you can get creative with how data is stored on the redis side and get the size to N.

For takehomes and in general people prefer readable code. Only optimize if it's needed. I mean I can write the whole thing in assembly code for ultimate performance. I can def see some nitpicker dinging me for optimized but unreadable code.

>The things you include/don't include by instinct I think is valuable signal.

Not to be insulting here. But I think your instincts are pretty off here. There's a limited amount of time assigned to this project. All the basics were included with what I did in the time allotted.

When you have good instincts you know when to apply automated testing, when to use error handling. These aren't things you should do all the time. Your attitude is something I see in many younger engineers. This attitude of rule following and best practices is just adhered to without thinking it all through properly. For the scope of the project. most of what you wrote here isn't needed.

You're also not testing what's traditionally tested in interviews. They test intelligence, depth knowledge and ability to design and solve problems. You're testing the ability to follow written rules. I mean following these things is trivial. It doesn't take much brain power to write error handling or unit tests. Perhaps you're just testing for someone that agrees with your programming philosophy. I mean these things can be followed regardless

Your mention of integration tests is telling. I mean the code for integration tests can rival the size of the code in the project itself with the a minor benefit in safety . It's not just take home assignments but many many many companies loaded with senior engineers skip over automated integration tests and keep those tasks in todo lists for the longest time. There's a huge engineering cost to building out testing infra to the maximum ideal, and many companies just don't bother. It's mostly the bigger well established companies that have the resources to do this.

>I'd disagree that it's worse than algo interviews, because algos are rarely the most important thing in a professional code base, especially one with that's about web services and not something more demanding.

It is worse in my opinion because it's biased. What if your opinion was more correct then my opinion but I judged you based off of my world view? Would that be right? No it wouldn't. Programmers are diverse and the best organizations have diversity in opinions and expertise.

All in all I mostly disagree but I think your right on one thing: At least one of those job candidate evaluators that are looking at my code likely had a similar attitude as you do and I should bias my code towards that more. Likely won't write integration tests though, hard no on that one.

Also good catch on the /test/-1/. That's definitely a legit ding imo.

> Yeah but many companies don't do integration testing as the infra on this is huge and writing tests is more complicated. It's certainly overkill for take home. I would say you're wrong here. Completely. This company was not expecting integration tests as part of the project at all. > > I've worked for start ups most of my career, and I'm applying to start ups. It's mostly the huge companies that have the resources to spend the effort to have full coverage on that. > > Integration tests are Not easy, btw. Infrastructure is not easy to emulate completely, how would I emulate google big query or say aws iot on my local machine? Can't, most likely integration tests involve actual infra, combined with meta code that manipulates docker containers.

I think I can agree with this -- it's true that most companies don't do it, but personally just spinning the thing up and throwing it a web request usually has rails in most languages these days. I spend more time in the NodeJS ecosystem, so I have things like supertest (https://www.npmjs.com/package/supertest) so maybe I'm spoiled.

> I feel this is such a minor thing. I'm sure most people would agree if you dinged that you'd be the one going overboard, not the candidate.

Yeah this is pretty reasonable -- opting in to typing at all is more a plus than a negative, on balance.

> You're going to build objects dynamically every execution Anyway. This saves you the extra intermediary step of not having to save it to an extra list.

It seems like I wasn't clear about how it's not about the lists/generators, so here's an explicit instance:

random_string_generator (https://github.com/anonanonme/takehome-sample/blob/master/ut...) is created as a callable every single time generate_test_paths. This is unnecessary, and could be pulled out to just be a static function.

If your argument is that the lambda gets optimized out, or that the benchmarked difference is insignificant (it very well could be!), then I could understand that.

> It's a reasonable assumption that all users assume /xx/xx equals /xx/xx/. It's also reasonable for a client to handle a 302 redirect. The specs didn't specify the exact definition on either of these things.

I'd argue the difference is so significant that it warranted a note in the documentation on the semantics, and if you google things like "trailing slash" it's been a thorn in peoples' sides for a long time.

Specs did specify test cases -- and none of them had a trailing slash.

I think we're really in the weeds here (in any reasonable working environment, this isn't a big deal), but it's wasteful to have every request become 2.

> So? logN is super fast. You have 100 objects binary or whatever indexed insert redis uses gets there in at most 4 or 5 jumps. So even if it's the hot path redis can take it. > > NLogN is slow enough that if N is large there is noticeable slow downs EVEN when it's a single call. Recall that redis is single threaded sync. NlogN can block it completely. > > Either way this is all opinion here right? Ideally the results are returned unsorted. the client is the best place to sort this, but that's not the requirement.

These are good points. Thinking about it though:

- logN is not faster than O(1) - NLogN is in the API, not redis -- redis experiences N while the scan is going

You're right that NLogN would certainly have a chance of blocking redis completely much more than N would, but I think in both cases redis experiences O(N) behavior for the stats endpoint, not NLogN.

> Of course the act of reading and writing data will cost N regardless. But your reasoning is incorrect. Look at that picture again: https://i.stack.imgur.com/osGBT.jpg. O(NlogN) will dwarf O(N) when N is large enough. It will even dwarf IO round trip time and has the potential of blocking redis completely on processing the sort.

Ahh, see the point about N/NLogN -- it's not Redis that experiences NlogN, it's the API. Redis just happily feeds all the values down, and they're aggregated at the python level.

Now, as far as the python level goes, I'm arguing that NLogN for in-memory number is going to be tiny compared to doing network requests and serializing JSON.

> Sure but look at the specs. It's asking for http requests...

That's reasonable -- the assumption is basically that the return fits in one request. I do think not trying to download the whole world at the same time is a legitimate concern, but it's not necessarily a pressing one since it wasn't mentioned.

> I mean I could build a full logger and error handler framework here...

Ah see I would expect you to pull one in -- just like you know of poetry, I'd be impressed by seeing more well-built creature comfort tools like that. Knowing good off-the-shelf tooling is a plus, IMO.

I agree with you that this is probably not what they expected (it's only a 4 hour take home!) but just wanted to make the point.

> For takehomes and in general people prefer readable code. Only optimize if it's needed. ...

Yeah that's true, too much optimization would definitely be bad -- but my point wasn't that you should optimize, it was that the approach I was talking about could be optimized (so that would be my answer to the NLogN versus N questions).

I think my version is the least surprising one -- no one has to know about pipeline or worry about atomicity. Just an O(1) operation to redis, like most people would expect to see.

> Not to be insulting here. But I think your instincts are pretty off here. There's a limited amount of time assigned to this project. All the basics were included with what I did in the time allotted. >

I think maybe I wasn't clear -- your instinct is what's on display. What you choose to include/use/not use is valuable signal (and that's the point of the take home).

> When you have good instincts you know when to apply automated testing, when to use error handling. These aren't things you should do all the time...

Well, if making sure to include automated testing for APIs and error handling is junior, I look forward to staying a junior engineer for my whole life. I don't think I'd submit a take home coding test without tests.

I'm absolutely not dogmatic about it (and I have the repos to prove I don't always write tests :), but I certainly am not proud to show anyone untested code -- it is enshrined in my mind as a measure of quality. If I'm putting my best foot forward, the code will have tests, especially when it's languages with weak type systems.

If the prompt was "write this like you're at a startup that has no money and no time", then sure. But even then -- technical debt forces rewrites at startups all the time, and the often the effect is worse when people don't write tests. Most of the time, you move slow so you can move fast.

> You're also not testing what's traditionally tested in interviews. They test intelligence, depth knowledge and ability to design and solve problems. You're testing the ability to follow written rules. I mean following these things is trivial. It doesn't take much brain power to write error handling or unit tests. Perhaps you're just testing for someone that agrees with your programming philosophy. I mean these things can be followed regardless >

I think this is the difference between take homes and whiteboard algo tests. I think the point of the take home is to see what you will do, on a realistic project that you were given full control over. That's what makes it better than the algo tests IMO -- it's more realistic and you have full control.

Everyone knows they should write tests... But do you actually? Because one thing is definitely harder/takes more discipline than the other.

> Your mention of integration tests is telling. I mean the code for integration tests can rival the size of the code in the project itself with the a minor benefit in safety . It's not just take home assignments but many many many companies loaded with senior engineers skip over automated integration tests and keep those tasks in todo lists for the longest time. There's a huge engineering cost to building out testing infra to the maximum ideal, and many companies just don't bother. It's mostly the bigger well established companies that have the resources to do this.

Yes, but this is a small API -- you literally have to write a test that hits the server once. There are libs for doing this with flask, there is documentation showing you how. It's not rocket science, and it's crucial to catching bugs down the road.

> It is worse in my opinion because it's biased. What if your opinion was more correct then my opinion but I judged you based off of my world view? Would that be right? No it wouldn't. Programmers are diverse and the best organizations have diversity in opinions and expertise. > > All in all I mostly disagree but I think your right on one thing: At least one of those job candidate evaluators that are looking at my code likely had a similar attitude as you do and I should bias my code towards that more. Likely won't write integration tests though, hard no on that one. > > Also good catch on the /test/-1/. That's definitely a legit ding imo.

Just about everything with humans in the loop is biased -- that's the world we live in. I don't know if they tried to run your test, but it could have failed with a 3xx and then they disqualified you right there, or bad input or whatever.

But yeah unfortunately it's hard to learn anything from this other than giving opinions on what rang bells for me.

Would be awesome if they gave you feedback.

Overall the code is reasonable and probably works for the normal cases -- it is a mystery why they wouldn't give feedback.

>If your argument is that the lambda gets optimized out, or that the benchmarked difference is insignificant (it very well could be!), then I could understand that.

In interpreted languages I believe the thing that gets saved to mem is a pointer to the code itself. So no inherit difference here in terms of allocation. For closures reference counting is increased for the referenced variables but this is not a closure.

Still this is a minor thing. Scoped functions are used for structural purposes performance benefits of using or not using them are negligible.

>logN is not faster than O(1) - NLogN is in the API, not redis

My point is log(n) is comparable to o(1). It's that fast.

NlogN is not comparable to O(n). In fact if you look at that pic NlogN is more comparable to O(n^2).

You definitely don't want NlogN on the application server whether its nodejs or python. That will block the python or node thread completely if n is large.

It's preferable to use SORT in redis then in python or node because redis is highly optimized for this sort of thing (punintended) as it's written in C. Even if its not in redis, in general for backend web development you want to move as much compute towards the database and off the webserver. The webserver is all about io and data transfer. The database is where compute and processing occurs. Keep sorting off web servers and leave it all in the database. You're a nodejs dev? This is more important for node given that the default programming model is single threaded.

Overall it's just better to use sets with scores because logN is blazingly fast. It's so fast that for databases it's often preferential to use logN tree based indexing over hashed indexing even when hashed indexing is appropriate.

Yes hashed indexing are O(1) average case insert and read but the memory savings of tree based indexing overshadows the negligible logN cost.

>I think my version is the least surprising one -- no one has to know about pipeline or worry about atomicity. Just an O(1) operation to redis, like most people would expect to see.

Two things here.

1st. Sorting on the web API is definitively wrong. This is especially true in nodejs where the mantra is always non blocking code.

Again the overall mantra for backend web is for compute to happen via the database.

2nd. Pipeline should 100 percent be known. Atomic operations are 100 percent required across shared storage and it's expected to occur all the time. Database transactions are fundamental and not unexpected. Pipeline should be extremely common. This is not an obscure operation. I commented about it in my code in case the reviewer was someone like you who isn't familiar with how popular atomic database transactions are, but this operation is common it is not a obscure trick.

Additionally pipeline has nothing to do with your or my own implementation. It's there to make atomic a state mutation and retrieval of that state.

I add one to a score then I retrieve the score and I want to make sure no one changes the score in between the retrieval and the add one. This is needed regardless of what internal data structure is used in redis.

>I don't know if they tried to run your test, but it could have failed with a 3xx

That'd be stupid in my opinion. 302 redirect is something I expected. Maybe I should have stated that explicitly in the docs.

>IMO -- it's more realistic and you have full control.

Unlikely. Even as a lead it's better to cater to the majority opinion of the team. 99 percent of the time you never have full control.

Developers and leads typically evolve to fit into the overall team culture while inserting a bit of their own opinions.

I'm a rust and Haskell developer applying for a python job. Do I apply my entire opinion to control the team? No, I adapt to all the bad (in my opinion) and good practices of the team. I introduce my own ideas slowly and where appropriate.

> it is a mystery why they wouldn't give feedback.

This is easy. It's not a mystery. It's because they don't give a shit about me. They don't want to hire me so the business relationship is over. Legally they owe me zero time to explain anything and it's more efficient for the business to not waste time on feedback.

>Yes, but this is a small API -- you literally have to write a test that hits the server once. There are libs for doing this with flask, there is documentation showing you how. It's not rocket science, and it's crucial to catching bugs down the road.

No there isn't. You have to spin up redis. Flask documentation doesn't talk about that. Integration testing involves code that controls docker-compose. It's a bunch of hacky scripts with external process calls that are needed to get integration tests working.

It's not rocket science but not within the scope of a take-home. Additionally hacking with docker compose is not sustainable or future proof, eventually you will hit infra that can't be replicated with docker.

I will probably do it next time just to cater to people who share your viewpoints.

>If the prompt was "write this like you're at a startup that has no money and no time", then sure.

Lol. The prompt was write it in four hours ( no time) and they aren't paying me for it ( no money).

>Specs did specify test cases -- and none of them had a trailing slash.

What? In the specs all examples have trailing slashes. All of them.

https://github.com/anonanonme/takehome-sample/blob/master/RE...

Take a look again, you remembered wrong.

What the specs didn't specify was what to do when there is no trailing slash. So why not a 302? The client can choose what to do here. Either follow the redirect or treat it as an error.

> In interpreted languages I believe the thing that gets saved to mem is a pointer to the code itself. So no inherit difference here in terms of allocation. For closures reference counting is increased for the referenced variables but this is not a closure. ...

This has all been tread over before -- I won't rehash it since it seems like we're going in circles.

The hot path is worth optimizing first, in my mind -- giving up perf there for stats which will be called much less doesn't make sense to me.

To each their own!

> 1st. Sorting on the web API is definitively wrong. This is especially true in nodejs where the mantra is always non blocking code.

This assertion is so wide that it cannot be true. No matter what language you're in, you can absolutely do sorting in the API layer.

> 2nd. Pipeline should 100 percent be known. Atomic operations are 100 percent required across shared storage and it's expected to occur all the time. Database transactions are fundamental and not unexpected. Pipeline should be extremely common. This is not an obscure operation. I commented about it in my code in case the reviewer was someone like you who isn't familiar with how popular atomic database transactions are, but this operation is common it is not a obscure trick.

You don't need to pipeline if the operation is O(1) -- it's not a question of whether transactions are good/bad, I don't know why you're assuming people don't know about them.

It's the question of taking in unnecessary complexity here when there's an O(1) operation available. You're optimizing for an operation that is not on the hot path. The reason you have to do the atomic operation is because you're using an access pattern that requires it.

I know why and how you used an atomic transaction -- I disagree on whether it was necessary.

> That'd be stupid in my opinion. 302 redirect is something I expected. Maybe I should have stated that explicitly in the docs. > > What? In the specs all examples have trailing slashes. All of them.

This is correct, I read wrong and for that I apologize, they do have all trailing slashes. I was thinking of "/api/".

> Unlikely. Even as a lead it's better to cater to the majority opinion of the team. 99 percent of the time you never have full control. > > Developers and leads typically evolve to fit into the overall team culture while inserting a bit of their own opinions. >

I referred to the context of the project -- you did* have full control.

If you were thinking of it in a team context, you should have asked more clarifying questions and found out more of what they wanted or would look for, similar to a real team context.

> No there isn't. You have to spin up redis. Flask documentation doesn't talk about that. Integration testing involves code that controls docker-compose. It's a bunch of hacky scripts with external process calls that are needed to get integration tests working. > > It's not rocket science but not within the scope of a take-home. Additionally hacking with docker compose is not sustainable or future proof, eventually you will hit infra that can't be replicated with docker. > > I will probably do it next time just to cater to people who share your viewpoints.

Again, this is a matter of what people think is hard and what isn't. Integration testing does not require that you spin up docker compose -- I literally linked to a libraries that do this in a reusable (importable) way w/ docker.

You said you don't do this kind of testing regularly, so it is understandable that it's not easy for you.

If docker was the problem, you can open subprocesses and assume that redis is installed in the test environment (i.e. your laptop, or CI environment), or take a binary to the redis binary path from ENV.

You can even remove the burden of running redis from the test suite and just make sure it's running manually or in test suite invocation/setup. There are many options, and it's just not that hard in 2023.

Don't know what (if anything) turned them off, so can't say this matters -- but it would matter to me, at the very least it would be a large plus for people who found the time to implement it.

> Lol. The prompt was write it in four hours ( no time) and they aren't paying me for it ( no money).

This is true, you did not have much time and they weren't paying you so I guess the situations are similar.

>The hot path is worth optimizing first, in my mind -- giving up perf there for stats which will be called much less doesn't make sense to me.

You aren't getting what I'm saying. logN is already a huge optimization for a hot path. Look at that pic: https://dev-to-uploads.s3.amazonaws.com/i/ya7xk4n9ch50xgwibt...

Seriously look at O(logN) and look at NlogN. Databases everywhere already take logN as acceptable. You are optimizing past that and introducing an NlogN sort step that will block all hot path calls because redis is single threaded. I don't think it's a to each their own thing here, i believe the user experience metrics will definitively play out towards my methodology but we can't know until we actually have metrics so we're at an impasse here.

>This assertion is so wide that it cannot be true. No matter what language you're in, you can absolutely do sorting in the API layer.

No this is absolutely true. Especially if you're a nodejs programmer. Nodejs is single threaded, sorting imposes NlogN penalty on a single thread.

Backend web development for the api layer is all about transmitting as much compute as possible to the database layer. You can get away with small sorts and things can still work but in general the practice of backend optimization is indeed to let the database handle as much as possible.

We can debate it but from my experience I know I'm pretty much categorically right on this. It's a fundamental principle. And it has special application to things that use Async IO like NodeJS. That's why there's huge emphasis in nodejs in writing non-blocking code. A sort is a potential block.

There are very few cases where you would handle compute on the api server rather then the database. The first instance is obvious, if N is guaranteed to be small which is not the case in the takehome. In other instances of sort with large N, the server ideally needs to launch that sort in a separate non-blocking thread so that the server can continue handling oncoming requests. In nodejs, to my knowledge since I last used it, you can't easily spawn a thread at will, you just have to pray you have available workers that aren't blocked by a sorting operation. In python the GIL makes this more complicated, but slightly better then nodejs.

For node you probably have 8 workers on 8 CPU cores. If you have 100 requests/s, all you need is 8 of them calling a sort on large N and all threads become blocked.

Don't believe me? Read the official docs of nodejs: https://nodejs.org/en/docs/guides/dont-block-the-event-loop#...

NlogN stands at the cusp here. It could work for small N, but for large N it will fail. Anything above NlogN is a huge risk.

>You don't need to pipeline if the operation is O(1) -- it's not a question of whether transactions are good/bad, I don't know why you're assuming people don't know about them.

This is categorically false. The pipeline operation as I said HAS NOTHING to do with O(1) or O(N).

This is essentially the operation:

    set["key"] += 1
    # another thread can update the value here leading to print displaying something unexpected. 
    print(set["key"])

So to solve the above comment you have to place both operations in a transaction. No two ways about it. The above operation is in spirit what I am doing. I update a value, then I read the value. Two operations and a possible race condition in between.

>Again, this is a matter of what people think is hard and what isn't. Integration testing does not require that you spin up docker compose -- I literally linked to a libraries that do this in a reusable (importable) way w/ docker.

You linked? Where? I'd like to know about any library that will do this. Tell me of any library that does integration tests that spins up infrastructure for you. The only one closest I can think of that you can run locally is anything that would use docker-compose or some other IAC language that controls containers. I honestly don't think any popular ones exist.

>If docker was the problem, you can open subprocesses and assume that redis is installed in the test environment (i.e. your laptop, or CI environment), or take a binary to the redis binary path from ENV.

No way I'm going to assume the user has redis installed on his local machine. Many devs don't. It's all remote development for them or everything lives in containers.

Docker is not the problem. Docker or virtual machines makes this problem more amenable to a solution, but even using docker here with testing is overkill and hacky. A take home should not expect a user to build excessive infrastructure locally just to run tests.

>You said you don't do this kind of testing regularly, so it is understandable that it's not easy for you.

I'd like to know your perspective. How is spinning up infrastructure for integration tests easy? You have to write code that launches databases, launches servers and views the entire infra externally. Do-able? Yes? Easy? depends on what you call easy. As easy as unit tests? Definitely no. Easy enough for a take home? In my opinion, no.

Maybe for this take home project you could be right. I could do some integration tests by spinning up docker-compose from within python. Hacky but doable. But in general this solution is not production scalable as production involves more things then what can be placed inside a docker-compose.

>You can even remove the burden of running redis from the test suite and just make sure it's running manually or in test suite invocation/setup. There are many options, and it's just not that hard in 2023.

So I should expect the reviewer to have redis running on his local machine? I would expect in 2023, I should have the entire project be segregated from the context it's running in. That's the cutting edge I believe.

Yeah it's 2023, you tell me how integration testing should be done as easily as unit testing on a takehome. I had one takehome project involving S3 and aws lambdas. They expected me to get an AWS account and literally set up infrastructure because there's no way around even testing what I wrote without actual infrastructure. That entire project was just integration test nightmare. Much rather run asserts locally.

>I referred to the context of the project -- you did* have full control.

Well I mean 99% of hires get into a context where they aren't in full control. So why is it logical to test what a developer will do if he/she had full control? Isn't it better to test the developers ability to code and to adapt to different contexts? Your methodology sort of just tests the programmers personal philosophy and whether it matches your own. That was my point.

> You aren't getting what I'm saying. logN is already a huge optimization for a hot path. Look at that pic: https://dev-to-uploads.s3.amazonaws.com/i/ya7xk4n9ch50xgwibt... > > Seriously look at O(logN) and look at NlogN. Databases everywhere already take logN as acceptable. You are optimizing past that and introducing an NlogN sort step that will block all hot path calls because redis is single threaded. I don't think it's a to each their own thing here, i believe the user experience metrics will definitively play out towards my methodology but we can't know until we actually have metrics so we're at an impasse here.

Just to keep the comparison in line -- we're comparing O(log(N)) @ O(1) in redis-land.

On the stats side we're comparing O(N) on the redis side PLUS either O(Nlog(N)) or O(N) on the python side.

> No this is absolutely true. Especially if you're a nodejs programmer. Nodejs is single threaded, sorting imposes NlogN penalty on a single thread.

NodeJS has both child processes and workers for heavy computation:

- https://nodejs.org/api/child_process.html

- https://nodejs.org/api/worker_threads.html

If you want to parallelize work on a thread pool there are libs that do just that:

- https://www.npmjs.com/package/piscina

((this is also part of the reason I think NodeJS is strictly superior to other "scripting" languages as mentioned briefly in the other comment))

Again, here you seem to be arguing against a strawman that doesn't know that blocking the IO loop is bad. Try arguing against one that knows ways to work around that. This is why I'm saying this rule isn't true. Extensive computation on single-threaded "scripting" languages is possible (and even if it wasn't, punt it off to a remote pool of workers, which could also be NodeJS!).

All of this is a premature optimization, but my point here is that the stats can be optimized later. You could literally compute it periodically or continuously and only return the latest cached version w/ a timestamp. This makes the stats endpoint O(1).

> This is categorically false. The pipeline operation as I said HAS NOTHING to do with O(1) or O(N). > ...

I think this is where we're talking past each other, so let me explain more of how I see the problem -- the solution I have in mind is serializing the URL and using ONE call to INCR (https://redis.io/commands/incr/) on the ingest side.

There is a lot you can do with the data storage pattern to make other operations more efficient, but on the stats side, the most basic way you can do it is to scan

I will concede that given that we know the data should fit in memory (otherwise you just crash) your approach gives you O(N) retrieval time and it's definitely superior to not have to do that on the python side (and python just streaming the response through). I am comfortable optimizing in-API computation, so I don't think it's a problem.

Here's what I mean -- you can actually solve the ordering problem in O(N) + O(M) time by keeping track of the max you've seen and building a sparse array and running through every single index from max to zero. It's overkill, but it's generally referred to as a counting sort:

https://ebrary.net/81651/geography/sorting_algorithms

This is overkill, clearly, and I will concede that ZSET is lighter and easier to get right than this.

> You linked? Where? I'd like to know about any library that will do this. Tell me of any library that does integration tests that spins up infrastructure for you. The only one closest I can think of that you can run locally is anything that would use docker-compose or some other IAC language that controls containers. I honestly don't think any popular ones exist.

https://testcontainers-python.readthedocs.io/en/latest/

I am very sure that I linked that, but in the case I didn't, here it is again -- hope you find it useful.

> No way I'm going to assume the user has redis installed on his local machine. Many devs don't. It's all remote development for them or everything lives in containers. > > Docker is not the problem. Docker or virtual machines makes this problem more amenable to a solution, but even using docker here with testing is overkill and hacky. A take home should not expect a user to build excessive infrastructure locally just to run tests.

I don't think these statements make sense -- having docker installed and having redis installed are basically equivalent work. At the end of the day, the outcome is the same -- the developer is capable of running redis locally. Having redis installed on your local machine is absolutely within range for a backend developer.

Also, remote development is not practiced by many companies -- the only companies I've seen doing thin-clients that are large.

> Maybe for this take home project you could be right. I could do some integration tests by spinning up docker-compose from within python. Hacky but doable. But in general this solution is not production scalable as production involves more things then what can be placed inside a docker-compose.

I see it as just spinning up docker, not compose -- you already have access to the app (ex. if it was buildable via a function) so you could spawn redis in a subprocess (or container) on a random port, and then spawn the app.

I agree that it is not trivial, but the value is high (in mymind.

> Yeah it's 2023, you tell me how integration testing should be done as easily as unit testing on a takehome. I had one takehome project involving S3 and aws lambdas. They expected me to get an AWS account and literally set up infrastructure because there's no way around even testing what I wrote without actual infrastructure. That entire project was just integration test nightmare. Much rather run asserts locally.

I agree that integration testing is harder -- I think there's more value there.

Also, for replicating S3, minio (https://github.com/minio/minio) is a good stand-in. For replicating lambda, localstack (https://docs.localstack.cloud/user-guide/aws/lambda/) is probably reasonable there's also frameworks with some consideration for this (https://www.serverless.com/framework/docs/providers/aws/guid...) built in.

That said I do think that's a weakness of the platform compute stuff -- it is inconvenient to test lambda outside of lambda.

> Well I mean 99% of hires get into a context where they aren't in full control. So why is it logical to test what a developer will do if he/she had full control? Isn't it better to test the developers ability to code and to adapt to different contexts? Your methodology sort of just tests the programmers personal philosophy and whether it matches your own. That was my point.

Ah, this is true -- but I think this is what people are testing in interviews. There is a predominant culture/shared values, and the test is literally whether someone can fit into those values.

Whether they should or should not be, that's at least partially what interviews are -- does the new team member feel the same way about technical culture currently shared by the team.

Now in the case of this interview your solution was just fine, even excellent (because you went out of your way to do async io, use newer/easier packaging methodologies, etc), but it's clearly not just that.

>Again, here you seem to be arguing against a strawman that doesn't know that blocking the IO loop is bad. Try arguing against one that knows ways to work around that. This is why I'm saying this rule isn't true. Extensive computation on single-threaded "scripting" languages is possible (and even if it wasn't, punt it off to a remote pool of workers, which could also be NodeJS!).

Very rare to find a rule that's absolutely true.. I clearly stated exceptions to the rule (which you repeated) but the generality is still true.

Threading in nodejs is new and didn't exist since the last time I touched it. It looks like it's not the standard use case as google searches still have websites with titles saying node is single threaded everywhere. The only way I can see this being done is multiple Processes (meaning each with a copy of v8) using OS shared memory as IPC and they're just calling it threads. It will take a shit load of work to make v8 actually multi-threaded.

Processes are expensive so you can't really follow this model per request. And we stopped following threading per request over a decade ago.

Again these are exceptions to the rule, from what I'm reading Nodejs is normally still single threaded with a fixed number of worker processes that are called "threads". Under this my general rule is still generally true: backend engineering does no typically involve writing non blocking code and offloading compute to other sources. Again, there are exceptions but as I stated before these exceptions are rare.

>Here's what I mean -- you can actually solve the ordering problem in O(N) + O(M) time by keeping track of the max you've seen and building a sparse array and running through every single index from max to zero. It's overkill, but it's generally referred to as a counting sort:

Oh come on. We both know these sorts won't work. These large numbers will throw off memory. Imagine 3 routes. One route gets 352 hits, another route gets 400 hits, and another route gets 600,000 hits. What's Big Oh for memory and sort?

It's O(600,000) for both memory and runtime. N=3 and it doesn't even matter here. Yeah these types of sorts are almost never used for this reason, they only work for things with smaller ranges. It's also especially not useful for this project. Like this project was designed so "counting sort" fails big time.

Also we don't need to talk about the O(N) read and write. That's a given it's always there.

>I don't think these statements make sense -- having docker installed and having redis installed are basically equivalent work. At the end of the day, the outcome is the same -- the developer is capable of running redis locally. Having redis installed on your local machine is absolutely within range for a backend developer.

Unfortunately these statements do make sense and your characterization seems completely dishonest to me. People like to keep their local environments pure and segregated away from daemons that run in a web server. I'm sure in your universe you are claiming web developers install redis, postgresql and kafka all locally but that just sounds absurd to me. We can agree to disagree but from my perspective I don't think you're being realistic here.

>Also, remote development is not practiced by many companies -- the only companies I've seen doing thin-clients that are large.

It's practiced by a large amount and basically every company I've worked at for the past 5 years. Every company has to at least partially do remote dev in order to fully test E2E stuff or integrations.

>I see it as just spinning up docker, not compose -- you already have access to the app (ex. if it was buildable via a function) so you could spawn redis in a subprocess (or container) on a random port, and then spawn the app.

Sure. The point is it's hacky to do this without an existing framework. I'll check out that library you linked.

>I agree that integration testing is harder -- I think there's more value there.

Of course there's more value. You get more value at higher cost. That's been my entire point.

>Also, for replicating S3, minio (https://github.com/minio/minio) is a good stand-in. For replicating lambda, localstack (https://docs.localstack.cloud/user-guide/aws/lambda/) is probably reasonable there's also frameworks with some consideration for this (https://www.serverless.com/framework/docs/providers/aws/guid...) built in.

Good finds. But what about SNS, IOT, Big Query and Redshift? Again my problem isn't about specific services, it's about infra in general.

>Ah, this is true -- but I think this is what people are testing in interviews. There is a predominant culture/shared values, and the test is literally whether someone can fit into those values.

No. I think what's going on is people aren't putting much thought into what they're actually interviewing for. They just have some made up bar in their mind whether it's a leetcode algorithm or whether the guy wrote a unit test for the one available pure function for testing.

>Whether they should or should not be, that's at least partially what interviews are -- does the new team member feel the same way about technical culture currently shared by the team.

The answer is no. There's always developers who disagree with things and just don't reveal it. Think about the places you worked at. Were you in total agreement? I doubt it. A huge amount of devs are opinionated and think company policies or practices are BS. People adapt.

>Now in the case of this interview your solution was just fine, even excellent (because you went out of your way to do async io, use newer/easier packaging methodologies, etc), but it's clearly not just that.

The testing is just a game. I can play the game and suddenly I pass all the interviews. I think this is the flaw with your methodology as I just need to write tests to get in. Google for example in spirit attempted another method which involves testing IQ via algorithms. It's a much higher bar

The problem with google is that their methodology can also be gamed but it's much harder to game it and often the bar is too high for the actual job the engineer is expected to do.

I think both methodologies are flawed, but hiring via ignoring raw ability and picking people based off of weirdly specific cultural preferences is the worse of the two hiring methodologies.

Put it this way. If a company has a strong testing culture, then engineers who don't typically test things will adapt. It's not hard to do, and testing isn't so annoying that they won't do it.

> Very rare to find a rule that's absolutely true.. I clearly stated exceptions to the rule (which you repeated) but the generality is still true. > > Threading in nodejs is new and didn't exist since the last time I touched it. It looks like it's not the standard use case as google searches still have websites with titles saying node is single threaded everywhere. The only way I can see this being done is multiple Processes (meaning each with a copy of v8) using OS shared memory as IPC and they're just calling it threads. It will take a shit load of work to make v8 actually multi-threaded. > > Processes are expensive so you can't really follow this model per request. And we stopped following threading per request over a decade ago. > > Again these are exceptions to the rule, from what I'm reading Nodejs is normally still single threaded with a fixed number of worker processes that are called "threads". Under this my general rule is still generally true: backend engineering does no typically involve writing non blocking code and offloading compute to other sources. Again, there are exceptions but as I stated before these exceptions are rare.

You seem to always fight the least knowledgeable strawmen. Process pooling and task distribution exist. Just because you didn't know about worker threads in NodeJS has nothing to do with me. Whether it's common or not has nothing to do with the validity of the solution.

Yet another reason why NodeJS is superior as a platform to CPython.

Anyway, I will take that you have relaxed your rule -- it needed to be relaxed. IIRC you said it was "definitively true" or something to that effect.

> Oh come on. We both know these sorts won't work. These large numbers will throw off memory. Imagine 3 routes. One route gets 352 hits, another route gets 400 hits, and another route gets 600,000 hits. What's Big Oh for memory and sort?

The array is sparse, and yes, this is exactly what I expect to happen. I wrote the scenario thinking of that.

> It's O(600,000) for both memory and runtime. N=3 and it doesn't even matter here. Yeah these types of sorts are almost never used for this reason, they only work for things with smaller ranges. It's also especially not useful for this project. Like this project was designed so "counting sort" fails big time. > > Also we don't need to talk about the O(N) read and write. That's a given it's always there.

Modern processors churn through numbers pretty quick, 600,000 operations is not a big deal -- still O(N)!

It's better to be clear than unclear -- it was a cause for confusion, evidently because you were comparing O(Log(N)) and O(NLog(N)). The comparison for ingest O(Log(N)) and O(1).

This is the last I'm going to say on this.

> Unfortunately these statements do make sense and your characterization seems completely dishonest to me. People like to keep their local environments pure and segregated away from daemons that run in a web server. I'm sure in your universe you are claiming web developers install redis, postgresql and kafka all locally but that just sounds absurd to me. We can agree to disagree but from my perspective I don't think you're being realistic here.

In the world before docker and docker compose, this is what people did when they had to run software to test locally. In the current world, people still do this, but it's made easier by docker and docker-compose.

That's the last I'm going to say on that.

> Sure. The point is it's hacky to do this without an existing framework. I'll check out that library you linked.

God forbid we write useful code without an existing framework.

> Of course there's more value. You get more value at higher cost. That's been my entire point.

Except "cost" here is relative -- not everyone considers that cost to be high. Relative to unit tests it is higher, but it is still trivial with knowledge.

It's simple, you just didn't have the knowledge so you thought it was prohibitively hard. That's fine -- that's just how technology goes.

> Good finds. But what about SNS, IOT, Big Query and Redshift? Again my problem isn't about specific services, it's about infra in general.

The goalposts seem to be moving, so time for me to get off the field. You noted those other tech specifically, so I answered.

Read up on ways you can replicate these environments locally, or don't. If you have some you can't replicate, don't use them, or find other ways to test them that aren't completely local (i.e. running in a test account).

That's the last I'm saying on that.

> No. I think what's going on is people aren't putting much thought into what they're actually interviewing for. They just have some made up bar in their mind whether it's a leetcode algorithm or whether the guy wrote a unit test for the one available pure function for testing.

Alright well I hope you're able to find a way through the minefield of reality, either way that's the last I'm saying on it.

> The testing is just a game. I can play the game and suddenly I pass all the interviews. I think this is the flaw with your methodology as I just need to write tests to get in. Google for example in spirit attempted another method which involves testing IQ via algorithms. It's a much higher bar > > The problem with google is that their methodology can also be gamed but it's much harder to game it and often the bar is too high for the actual job the engineer is expected to do. > > I think both methodologies are flawed, but hiring via ignoring raw ability and picking people based off of weirdly specific cultural preferences is the worse of the two hiring methodologies. > > Put it this way. If a company has a strong testing culture, then engineers who don't typically test things will adapt. It's not hard to do, and testing isn't so annoying that they won't do it.

Well, don't expect to win the game if you don't play.

Hope this was illustrative of at least how others think -- this is the last I'm going to post, thanks for the discussion!

>Well, if making sure to include automated testing for APIs and error handling is junior, I look forward to staying a junior engineer for my whole life. I don't think I'd submit a take home coding test without tests.

Error handling is already done by the framework. If you want explicit handles that's just busy work. Most python crashes involve traceback sent to stderr telling you what error occured and returns a 500. which is exactly the kind of handling I want. I'd just be replicating the framework code with the same logic everywhere if I did what you mentioned.

I'll give you credit for the unit tests. There's only 2 unit testable functions in the entire code and it's all in utils.py. Practically speaking only the random path generator is worth any sort of automation testing, and the http request callee is io, so not unit testable. Integration tests are raw overkill on takehomes. Manipulating docker compose in test code is just overboard.

Are you currently a junior? If you are then the insight you should eventually achieve is that unit tests for web application servers is not as important as you currently think.

This is because web servers are primarily io based applications. There's overall not much pure logic you can abstract to unit tests.

Pure logic rarely has errors as it doesn't deal with state. You will find that unit tests become a lot of extra busy work. If your pure logic minimizes mutations and you properly use typing and use functional code your error rate will drop to the point where you mostly write unit tests for other people. Writing it for your own code is essentially writing thousands of lines only to find it didn't catch a single error. It's still good to have unit tests but the cost benefit analysis becomes out of whack the majority of the time.

Did you know Edward Dijkstra was famously against testing? He promoted a different sort of art that would guarantee your code worked 100 percent of the time for all cases. Say I had a unit test: assert(add(100, 1) 101). That only covers one test case out of trillions of possible errors. His method would cover the entire spectrum of possible cases without a single test.

Basically a good coder should be hybridized in his approach. Code in a way that uses Dijkstra's philosophy such that the need for tests are minimized then write the tests you need without excessive focus on it because you should know that your tests are only covering a fraction of possible test cases anyway.

Not all engineers discover this method or even take the time to internalize and utilize it. Thus their code often ends up being error prone enough to justify unit testing or test driven development. Many super senior engineers don't even know about this. The staff position is more of a political title that has to do with leadership and architectural planning.

Anyway, unique to backend engineering... Integration tests are where most errors occur. It's where state is mutated and where race conditions, dead lock and where most compute logic is directed towards. It's 100x more important to focus on testing at the integration level.

The problem here though is, like I said, the infra around integration tests is huge. So often it doesn't get done and you see a bunch of companies just end up using zero integration tests and some small set of unit tests that are mostly inconsequential.

>but I certainly am not proud to show anyone untested code -- it is enshrined in my mind as a measure of quality

Deshrine this if you can. There are many techniques outside of this single methodology of testing for writing good code.

For example, have you heard of Idris? Idris is a language where you can use the type system to prove the program is correct. Test cases only cover correctness for a specific case. For Idris you can write a sorting algorithm and literally prove it to be correct for all cases without writing a single test.

I'm not a complete adherent to total proof based methodologies. As mentioned previously I follow a hybrid approach such that my need for testing is minimized. Remember Enshrining any concept blinds you from alternative paths.

I view excessive unit tests as a crutch. Any programmer who actually relies on unit tests during programming to catch a lot of his own errors actually imo has a flawed model inside his head on how to compose logic while minimizing errors.

The experience of a web application programmer with a good model of logic composition in his head is that the unit tests should be passing asserts the overwhelming majority of the time.

If you are consistently using it to pick out hidden logic errors in your code it sort of works from a practical standpoint but you're not on the ideal path.

Unit tests are good, don't get me wrong but Enshrining the concept and relying on it, is a crutch.

>Everyone knows they should write tests... But do you actually? Because one thing is definitely harder/takes more discipline than the other.

They only do it if the entire team does it. Otherwise the majority of people don't do it in the face of looming deadlines. Unit tests are usually baked into dev culture, if standards require it they'll do it. Thus logically if unit testing is required it's not really that important to measure this aspect during the interview as they will write it regardless of whether or not they do it in the interview.

> Error handling is already done by the framework. ...

You literally had a bug in your code that would have been caught with testing, and it did not have to do with the logic but with input sanitization, which has to do with your integration with what the framework provided you.

I am not advocating for unit tests, I was advocating for any tests at all, in the code sample. I already said that the highest value tests in my mind are E2E tests. The ones that make sure your code does what it says it does, in as black-box a way as possible.

Also, I want to note that the logic you're applying works best with languages with strong type systems, not Python.

I don't know what your point was on not testing is (were you trying to allude to property testing/fuzzing?), but here in the real world, tests are good for testing edge cases and preventing regressions.

I am only comfortable skipping tests completely in languages with strong, expressive type-systems -- Typescript, Rust and Haskell. Even then, I still make sure to write regression tests and think of what can go wrong.

Again, YOU think that the difficulty to writing integration tests is huge, and that's because you don't do it a lot. To me that's a liability, not something I want to adopt -- the faster I can write tests and prevent regressions, the better code I am able to write, the easier it is to integrate my code and have confidence that changes don't break existing functionality.

> Deshrine this if you can. There are many techniques outside of this single methodology of testing for writing good code.

Absolutely not -- no matter how strong your type system is, it will bend and break during contact with the real world. E2E tests, regression tests and the like are crucial for good engineering.

> For example, have you heard of Idris? Idris is a language where you can use the type system to prove the program is correct. Test cases only cover correctness for a specific case. For Idris you can write a sorting algorithm and literally prove it to be correct for all cases without writing a single test. > > I'm not a complete adherent to total proof based methodologies. As mentioned previously I follow a hybrid approach such that my need for testing is minimized. Remember Enshrining any concept blinds you from alternative paths. > > I view excessive unit tests as a crutch. Any programmer who actually relies on unit tests during programming to catch a lot of his own errors actually imo has a flawed model inside his head on how to compose logic while minimizing errors. > > The experience of a web application programmer with a good model of logic composition in his head is that the unit tests should be passing asserts the overwhelming majority of the time. > > If you are consistently using it to pick out hidden logic errors in your code it sort of works from a practical standpoint but you're not on the ideal path. > > Unit tests are good, don't get me wrong but Enshrining the concept and relying on it, is a crutch.

You seem to be battling one heck of a strawman here. I have not said that you should write excessive unit tests. I said that you should have at least any tests, and that I find the most value in E2E tests.

Also, Idris is basically Haskell w/ dependent types, but what you're saying about it is full of half-truths. You cannot use the type system to prove a program is correct, you use type systems to prove a program is sound. Correctness is problem dependent, input dependent, and many other things that are situation-dependent.

With a strong type system, you can avoid unit tests, but only if you use that type system properly (ex. using a Natural instead of an Integer where it is appropriate).

Regardless of whether you have a strong type system or not, tests (of all kinds, but especially E2E and regression) are valuable and have their place.

> They only do it if the entire team does it. Otherwise the majority of people don't do it in the face of looming deadlines. Unit tests are usually baked into dev culture, if standards require it they'll do it. Thus logically if unit testing is required it's not really that important to measure this aspect during the interview as they will write it regardless of whether or not they do it in the interview.

I think it's a matter of culture. Are you the person that makes time to write these/makes it part of your plan, or not?

There are people who do, and people who don't.

>You literally had a bug in your code that would have been caught with testing, and it did not have to do with the logic but with input sanitization, which has to do with your integration with what the framework provided you.

No this bug wouldn't have been caught. I'd have to write the test for it.

I did do integration tests. I just did them manually. If I wrote all the manual tests and made them automated this "error" would still be there. It's a matter of luck whether I manually test or write the integration test that finds this error.

It's also technically not huge bug. The server returns a 500 error instead of a 400 error.

>I am not advocating for unit tests, I was advocating for any tests at all, in the code sample. I already said that the highest value tests in my mind are E2E tests. The ones that make sure your code does what it says it does, in as black-box a way as possible.

Ok you got this. You're right on this. I should put in some testing here. even if it's minimal.

>Also, I want to note that the logic you're applying works best with languages with strong type systems, not Python.

This is where you are wrong. Python has what I call an opt-in type system meaning it's optional and it can be as strong as you want it to be. You choose the external type checker like mypy and those type checkers can be as powerful as you want.

The python type system can even do Haskell/Rust level exhaustive matching on sum types: https://tech.preferred.jp/en/blog/python-exhaustive-union-ma...

As you've seen with my JSON type, python allows for recursive types and sum types. Which makes it more versatile than golang which can't do either even though it has a "strong" type system. All the bells and whistles for a modern type system are there and more.

In fact I believe python may even exceed typescript in power. I believe type script won't do exhaustive checks on sum types.

>I am only comfortable skipping tests completely in languages with strong, expressive type-systems -- Typescript, Rust and Haskell. Even then, I still make sure to write regression tests and think of what can go wrong.

So modern python matches these language in type level power, it's just optional so it's not as pervasive as typescript but the option is there and when utilized it's equal if not better in power. But proof based coding isn't solely just on using types to prove out your code. I just used idris as an example of an alternative path. The reality is Idris is a bit too extreme because coding those types takes too much effort.

Basically i change my coding style to both minimize errors and allow me to build a minimal proof in my head. I use coding style, proofs in my head, and a strong/versatile type system all in conjunction to minimize error rate to low enough levels that tests are less important.

>Again, YOU think that the difficulty to writing integration tests is huge, and that's because you don't do it a lot. To me that's a liability, not something I want to adopt -- the faster I can write tests and prevent regressions, the better code I am able to write, the easier it is to integrate my code and have confidence that changes don't break existing functionality.

You can't do integration tests without using infrastructure and spinning up processes. Let me put it to you plainly. It's not ME who thinks it's difficult. It's the majority of the developer population knows it's much more difficult then unit tests. That's why such a segregation even exists. The fact that we categorize the tests into integration tests and unit tests speaks to huge differences between the two in difficulty.

You may think integration tests are easy, but that's just your own opinion. This is different from the overall general opinion which is more inline with what I'm saying.

>Absolutely not -- no matter how strong your type system is, it will bend and break during contact with the real world. E2E tests, regression tests and the like are crucial for good engineering.

Sure I can agree with your philosophy in spirit. Practically speaking though there's the effort involved in this that makes it so that many companies just don't do it.

Maybe it's changed recently, I've moved from backend web to embedded systems to arm devices and GPU programming so I've been away from web stuff for about 3 years. Generally no docker or as much infra in this universe. If docker-compose and mocking infra locally is universal on everyones local machine now, then yeah you could be right but from what I'm seeing... I don't think so.

Unless we have metrics I'm still going to go with the fact that in general it's not easy and most companies don't do it.

>You seem to be battling one heck of a strawman here. I have not said that you should write excessive unit tests. I said that you should have at least any tests, and that I find the most value in E2E tests.

Alright. I get what you're saying. Write tests, integration tests are the most important thing here. Unit tests are basically more or less practically useless for this takehome. And specific for this takehome I could've written integration tests here by writing some hack python script to call docker-compose. I agree.

Where we don't agree is the difficulty level of writing these tests and the general pervasiveness of integration or E2E tests in the industry overall. We can just leave it at that, without metrics the argument goes in circles. We've taken it as far as we could.

>Also, Idris is basically Haskell w/ dependent types, but what you're saying about it is full of half-truths. You cannot use the type system to prove a program is correct, you use type systems to prove a program is sound. Correctness is problem dependent, input dependent, and many other things that are situation-dependent.

No this is truth. You're just misinformed.

Dependent types are enough to prove a program is correct given a set of specifications. Of course not everything can be proven. But most things can. You can't prove the halting problem for example, but you can prove specs like determining if a list is sorted: https://dafoster.net/articles/2015/02/27/proof-terms-in-idri...

Also from the documentation of idris, (see the title): http://docs.idris-lang.org/en/latest/proofs/index.html

Theorem proving is the title. or aka proving anything mathematical or aka proving mathematical programs meet mathematical specs. Dependent typing is one way to go about this. There are other ways. The intended design of idris is similar to TLA+ the intention is to allow the user to prove the program correct to a spec and avoid unit tests via proof.

>With a strong type system, you can avoid unit tests, but only if you use that type system properly (ex. using a Natural instead of an Integer where it is appropriate).

How is using a nat more safe than an int? A nat just makes it more general.

I'm not 100% in agreement here. Let me add some nuance to my opinion. Unless the type system is more powerful then what you have in say haskell, it can only reduce the need of unit tests. You can't completely avoid it, but it can be reduced to the point where practically speaking you don't always need it. Certainly not for this takehome.

With something like idris you can, technically speaking, eliminate the need for unit tests all together. But this is a technical thing. In practice it's really hard to achieve this due to how hard it is to write these proofs. But the technical level of safety is operating at the highest level here.

>I think it's a matter of culture. Are you the person that makes time to write these/makes it part of your plan, or not?

Depends on the situation, the timelines, the brittleness of of the project and the technology. Highly, highly dependent on all sorts of critical variables. I'll tell you this: I don't enshrine the concept of testing. It's always a case by case basis.

If i had unlimited time though, I think your philosophy would be correct. Always write tests. But with limited time I artfully choose between a myriad of techniques (including choosing what to test) to meet certain timelines in the best way possible. This sometimes means not writing tests.

Also sometimes laziness is a factor, I'll be honest. If I'm confident my toy project or take home is correct I just won't write unit tests even if I have unlimited time. I think what I learned from this overall thread and from you is that I should always write a bunch of tests for the take home.

Hey I know this thread got a little heated. I want you know that I appreciate your viewpoints even though I disagree with a lot of points. I think heated but respectful debates are healthy and I definitely learned something here. Conversations where people are overly polite I feel like I learn nothing because people rather be polite then honest. So appreciate all the time you took to reply. I may be clashing hard against your points but there's nothing but respect here. You are clearly experienced.

> No this bug wouldn't have been caught. I'd have to write the test for it. ...

Ah OK, so this is where I think that we're talking past each other. When I say "testing", and the reason I focus on making them automated is that when I sit down to write tests, if I'm really up to it I get myself in a mood where I'm thinking of what can go wrong.

Doing that is what I'm suggesting might have made the difference.

> This is where you are wrong. Python has what I call an opt-in type system meaning it's optional and it can be as strong as you want it to be. You choose the external type checker like mypy and those type checkers can be as powerful as you want. ... > > The python type system can even do Haskell/Rust level exhaustive matching on sum types: https://tech.preferred.jp/en/blog/python-exhaustive-union-ma... >

So this comes back to me not writing much python -- but I have used typing + mypy briefly a while ago, but I'd need to see some claims it is nowhere near as powerful as Typescript/Rust/Haskell.

I'm not a python + mypy regular user though, so I don't know where it's limits are, but particularly with the haskell comparison, it does not support GADTs (https://github.com/python/mypy/issues/8252). Also, I know it doesn't support Rust's affine type system (ownership/borrows).

It's true that mypy could in theory do this, but it does not, and I do not expect it to anytime soon (backing for the other projects is more sound anyway).

I personally believe NodeJS is the best of the "scripting" languages, but that's a whole 'nother discussion.

> So modern python matches these language in type level power, it's just optional so it's not as pervasive as typescript but the option is there and when utilized it's equal if not better in power. But proof based coding isn't solely just on using types to prove out your code. I just used idris as an example of an alternative path. The reality is Idris is a bit too extreme because coding those types takes too much effort. > > Basically i change my coding style to both minimize errors and allow me to build a minimal proof in my head. I use coding style, proofs in my head, and a strong/versatile type system all in conjunction to minimize error rate to low enough levels that tests are less important.

I agree on being flexible, I like using Haskell/Rust because of this as well. I certainly disagree that python matches any of those others in type level power though.

> You can't do integration tests without using infrastructure and spinning up processes. Let me put it to you plainly. It's not ME who thinks it's difficult. It's the majority of the developer population knows it's much more difficult then unit tests. That's why such a segregation even exists. The fact that we categorize the tests into integration tests and unit tests speaks to huge differences between the two in difficulty. > > You may think integration tests are easy, but that's just your own opinion. This is different from the overall general opinion which is more inline with what I'm saying.

Regardless of what other devs thing of it, it's worth doing IMO. Here's the project I linked before

https://testcontainers-python.readthedocs.io/en/latest/READM...

> Dependent types are enough to prove a program is correct given a set of specifications. Of course not everything can be proven. But most things can.

Disagree with this. For example, memory leaks. If I define "correct" to be "doesn't leak memory", Haskell (for example -- don't know what the memory is like in Idris) is going to have a hard time.

There are things that are hard/impossible to encode into dependent types.

> Theorem proving is the title. or aka proving anything mathematical or aka proving mathematical programs meet mathematical specs. Dependent typing is one way to go about this. There are other ways. The intended design of idris is similar to TLA+ the intention is to allow the user to prove the program correct to a spec and avoid unit tests via proof.

Theorem proving != engineering, and this is why Haskell/Idris/ML languages at large never hit mainstream properly. If you try to write the kind of types that could even attempt to get correctness absolutely right, less than 0.1% of programmers would be able to use or work on your codebase.

I don't think TLA+ is relevant here because it's mostly used for testing distributed/concurrent systems in practice, and is not a PL so-to-speak.

> How is using a nat more safe than an int? A nat just makes it more general.

https://hackage.haskell.org/package/natural-numbers-0.1.2.0/...

Naturals (& peano numbers) are the categorical example of dependent types that are useful -- in Haskell, natural numbers are the set of positive integers including zero. It makes the -1 case impossible (i.e. fail to parse @ the program boundary), which is the point.

> With something like idris you can, technically speaking, eliminate the need for unit tests all together. But this is a technical thing. In practice it's really hard to achieve this due to how hard it is to write these proofs. But the technical level of safety is operating at the highest level here.

I think we're agreeing, but you just said two very different things -- most things (especially important ones) cannot be easily solved at the type level.

How do you type-check a redis connection at compile time? or a database like postgres at runtime? You can't, really -- because they are inherently dynamic things.

Now in the case of unit tests, I'd argue that a good type system is one of the only things that can prevent you from having to write lots of unit tests. I'm pretty sure we agree on this!

> I'll tell you this: I don't enshrine the concept of testing. It's always a case by case basis.

Yeah this is where we disagree -- I think this case called for it, you disagree.

We agree that sometimes tests are not necessary or warranted.

> Hey I know this thread got a little heated. I want you know that I appreciate your viewpoints even though I disagree with a lot of points. I think heated but respectful debates are healthy and I definitely learned something here. Conversations where people are overly polite I feel like I learn nothing because people rather be polite then honest. So appreciate all the time you took to reply. I may be clashing hard against your points but there's nothing but respect here. You are clearly experienced.

No problem, always glad to have a debate, and being able to have them on HN is why I still come to this site!

We can disagree, and I benefit from learning where your disagreements are (I don't there are actually many, I think our stances on testing and types are very similar outside of me just not liking in python in any meaningful way).

>Ah OK, so this is where I think that we're talking past each other. When I say "testing", and the reason I focus on making them automated is that when I sit down to write tests, if I'm really up to it I get myself in a mood where I'm thinking of what can go wrong. >Doing that is what I'm suggesting might have made the difference.

What? I get myself in the mood for manual tests too.

We're not talking past each other. You can't just quote "testing" and embed extra meaning into that after the fact. There's no way I would know what you're talking about.

>I'm not a python + mypy regular user though, so I don't know where it's limits are, but particularly with the haskell comparison, it does not support GADTs (https://github.com/python/mypy/issues/8252). Also, I know it doesn't support Rust's affine type system (ownership/borrows).

Nah when it comes to the lang extensions almost nothing comes close. I'm just talking about regular haskell ADTs.

The affine stuff isn't relevant as python uses a GC. What does it even mean to do borrow checking when nothing owns a variable in python?

It's quite obvious I'm just referring to the ADTs here.

Python supports regular ADTs better than typescript. So it's better imo. But that's just my initial opinion, I don't use ts extensively so it might be better.

>I personally believe NodeJS is the best of the "scripting" languages, but that's a whole 'nother discussion.

If You mean JS. then no, JS is horrible. Typescript has an argument here, I think that's what you mean.

>I agree on being flexible, I like using Haskell/Rust because of this as well. I certainly disagree that python matches any of those others in type level power though.

It matches it in terms of ADTs and where relevant.

>Theorem proving != engineering, and this is why Haskell/Idris/ML languages at large never hit mainstream properly. If you try to write the kind of types that could even attempt to get correctness absolutely right, less than 0.1% of programmers would be able to use or work on your codebase.

So? I never made this claim. I just said there are other methodologies to employ other then strict fanatical loyalty to testing. I simply used Idris as an example of an alternate path. In fact I specifically STATED that it would be hard to use proofs with Idris.

>I don't think TLA+ is relevant here because it's mostly used for testing distributed/concurrent systems in practice, and is not a PL so-to-speak.

The point was using proof to write correct programs. So it's completely relevant as TLA+ is used for that.

>Naturals (& peano numbers) are the categorical example of dependent types that are useful -- in Haskell, natural numbers are the set of positive integers including zero. It makes the -1 case impossible (i.e. fail to parse @ the program boundary), which is the point.

ah my mistake I had a brain fart and thought you were referring to Haskells Num or something along those lines. Yes if by Nat you mean Haskell Nat or unsigned int, then that makes sense.

>I think we're agreeing, but you just said two very different things -- most things (especially important ones) cannot be easily solved at the type level.

I never said two different things. I think you just made some assumptions and responded to some paragraphs without reading far enough. Like your "Theorem proving != engineering," comment. If you read further you'd realize I already know about what you're saying.

My overall point on this front is aspects of proving can help type checking and FP and other techniques reduce error rate by a large margin.

>How do you type-check a redis connection at compile time? or a database like postgres at runtime? You can't, really -- because they are inherently dynamic things.

I'm not sure why you're telling me this. Did I make a claim that type checking can work on a redis connection?

Do you remember when I said integration tests are more important then unit tests? And unit tests are mostly practically trivial? Well type checking is sort of involved with that. I mean these are points I've been making throughout our conversation suddenly your like "How do you type check a redis connection?" as if I haven't made any of those points. I made a point and you made the same point.

Anyway we agree here.

>Yeah this is where we disagree -- I think this case called for it, you disagree.

Yes. To get into more detail here. The point is the infra cost of integration tests we disagree on that, I don't think it's worth it all the time I think you believe it's worth it in more cases.

For the take home project I still don't think it's worth it. But I concede I'm wrong because the Reviewer is looking for tests and I need to cater to what the reviewer is looking for.

>We agree that sometimes tests are not necessary or warranted.

Well you told me you have testing enshrined. So I was disagreeing with that. I think you were just making a statement about the importance of testing. In actuality we're more in agreement about the nuance of testing.

>No problem, always glad to have a debate, and being able to have them on HN is why I still come to this site!

Sometimes these conversations generate some heat on HN. I actually don't mind the heat, keeps people brutally honest.

Great talking to you as well!

> What? I get myself in the mood for manual tests too.

> We're not talking past each other. You can't just quote "testing" and embed extra meaning into that after the fact. There's no way I would know what you're talking about.

Writing tests generally means... thinking about tests to write? If you think that thinking of edge cases is not a part of writing tests, then I don't know what to tell you.

I won't say any more on the subject.

> Nah when it comes to the lang extensions almost nothing comes close. I'm just talking about regular haskell ADTs. > > The affine stuff isn't relevant as python uses a GC. What does it even mean to do borrow checking when nothing owns a variable in python? > > It's quite obvious I'm just referring to the ADTs here. > > Python supports regular ADTs better than typescript. So it's better imo. But that's just my initial opinion, I don't use ts extensively so it might be better.

We're not considering language extensions (which are built in BTW, on par with using the stdlib), but we're considering completely separate ecosystem plugins that do type checking? Nonsense.

Tracking usage of variables is useful, whether you use GC or not -- it's a matter of ability in the type system. My point is that you cannot specify in your code that a value should be used "at most once", which is what affine types afford you.

I will not say more on this -- the point is absurd from the beginning, that mypy is as advanced as the literal PhD marsh that is Haskell or innovation that was Rust's borrow system for decades.

If you think mypy is better than Typescript, then we have nothing to talk about -- it's just unlikely for me to gain anything from that discussion.

> If You mean JS. then no, JS is horrible. Typescript has an argument here, I think that's what you mean.

No, I mean JS, and in particular NodeJS as an execution platform, because it has no GIL, can do threads, async io is a first class concept, were flexible enough to get used to transpilation (which lets something like Typescript exist).

> ah my mistake I had a brain fart and thought you were referring to Haskells Num or something along those lines. Yes if by Nat you mean Haskell Nat or unsigned int, then that makes sense.

The type is called Natural, and I wrote Natural.

> I'm not sure why you're telling me this. Did I make a claim that type checking can work on a redis connection?

You said that it can prove a system to be "correct". Unfortunately I can't know what you meant by "correct", but the type system will not help you with many of the practical issues that are most important when writing code.

That's where good engineering comes in.

This will be my last on this discussion, was good!

>I won't say any more on the subject.

Yeah don't bother.

>We're not considering language extensions (which are built in BTW, on par with using the stdlib), but we're considering completely separate ecosystem plugins that do type checking? Nonsense.

It's not nonsense, the language extensions should not be included because the functionality in Haskell is so far and above what ANY type checker typically does. Clearly.

You're being utterly too pedantic here. When I say python types match haskell in power I am obviously not touching upon GADT or RankN. You're getting into the weeds.

It has nothing to do with extensions being "on par with stdlib"

>Tracking usage of variables is useful, whether you use GC or not -- it's a matter of ability in the type system. My point is that you cannot specify in your code that a value should be used "at most once", which is what affine types afford you.

Yeah but not strictly necessary for python which has a GC, but strictly necessary for Rust which has ownership.

>I will not say more on this

Yeah don't bother. Sort of rude. But whatever.

>No, I mean JS, and in particular NodeJS as an execution platform, because it has no GIL, can do threads, async io is a first class concept, were flexible enough to get used to transpilation (which lets something like Typescript exist).

It can't do threads. I just looked it up. Worker Threads are processes that use some form of IPC. Literally a whole new v8 engine. I was right. Also JS is a horrible language. It's crazy you use haskell and you're ok with things like undefined which basically is a nonsense instance that can flow from one end of your code to another.

>The type is called Natural, and I wrote Natural.

Nope the type is called Nat. https://hackage.haskell.org/package/fin-0.3/docs/Data-Type-N... and in Idris: https://www.idris-lang.org/docs/current/base_doc/docs/Prelud....

But this has nothing to do with anything does it? I thought you were referring to a more general type Num.

>You said that it can prove a system to be "correct". Unfortunately I can't know what you meant by "correct", but the type system will not help you with many of the practical issues that are most important when writing code.

And I wrote extensively on what that means and you read it so you know what I'm talking about. There's no need to get pedantic and argue on pedantic points that are obviously contrary to the obvious meaning of what I'm saying.

>That's where good engineering comes in.

Yep, but do you have a point? Why make this statement?

>This will be my last on this discussion, was good!

I don't think you think it was good. I think you're annoyed. That's why you want to cut it off.

> Flask is an IO app. It's inherently not testable via unit tests because it's a server. You'd have to build integration tests around an entire server which is huge overkill for this project.

I disagree. You could argue that the type of tests outlined here https://flask.palletsprojects.com/en/2.3.x/testing/ are integration tests, in the sense that you're actually testing a full endpoint, but it doesn't require the kind of full integration testing setup that you're describing. I'm not suggesting this is a reason you "failed" the take-home — as others have mentioned, there's myriad reasons why you could have failed, none of which have anything to do with your code, but I did feel the need to correct what I see as a wrong understanding of the type of testing setup that is required here and what's possible.

There's a problem you're not seeing here. This solves it partially but in the end doesn't fix anything. The key thing here is redis is IO and it can't be tested.

Think of IO as a virus it pollutes the outer scope of anything it lies within. Anything that contains IO becomes IO and therefore is not unit testable. In fact any function marked with "await" or "async" is usually a good indicator of this pollution. Async await keywords actually spread around according to this exact logic I specified.

Example:

   async def handler(request) -> response:
         x = await iocall()
         y = add(2, x)
         return response(y)

iocall is in the scope of handler, handler becomes polluted with async and thus "not unit testable". I took the time to encapsulate pure addition logic into the add function, this makes add testable because it doesn't enclose over any IO functions.

That's the pattern of making unit testable code. You should try to move as much logic as possible into pure functions like add. Keep IO functions small and general and devoid of custom logic while segregating all your custom code into pure functions. Then just test these pure functions and leave everything else alone because all that other stuff goes into the realm of IO and integration tests.

You could get around this with mocking and using dependency injection but that comes with it's own set of problems including more complexity and spending the extra time to get the mocks to return results you want for each test. I would move these into the realm of integration tests rather then mock. In this case the usage of the flask test_client would still have to involve spinning up some sort of infra involving the database.

Your point about the unit test client from flask is completely valid for certain cases. It only works if the handler does not touch IO. In that case you wouldn't even need to mark the handler with the async keyword as nothing is awaited.

The payback on take home tests is seldom worth it. There is too much liability and risk and very little incentive to the company to provide honest feedback.

You should learn to accept the fact that you will not get an honest review. If you choose to enter in to a take home test go in to it with that understanding.

The interview process is a chance for them to review you, but also for you to review them. Make sure the process is equitable. If they ask for too much you owe it to yourself to push back. Ultimately they will respect you more if you stand up for yourself. It shows you have healthy boundaries and will protect their interests when you work for them.

Is there some case you can point out about liability risk of providing honest feedback?

I've heard this argument many times but can't quite get it. Based on my experience some companies do provide honest feedback, some don't. I try to ask in advance and decline those who will not give any feedback.

Sure. What if they are undecided about you. You're good but they want to consider other applicants. In that case you'll likely get silence.

But lets say they know you're not a fit. So they decline you based on your test project. What is the benefit to the hiring manager to have them or a developer sit down and tell you why you weren't a good fit? It costs them time a lot of time. That does not benefit their need, finding someone for their open role. The biggest negative to this behavior is a reputation hit for the company with that one person.

Liability: providing feedback gives you the interviewee something to argue about. It also is a communication done by acting members of the company and therefore relatively official. All it takes is someone being a bit rude, disrespectful, or even a misunderstanding to make this now official communication something somone could try and publically shame the company with. Its easier for them to say nothing or ver little at all.

With a third party recruiter there is an expectation that the recruiter will take a percentage of the placed applicants salary. And the employer expects the recruiter will bring them quality applicants. Therefore the rectuiter often has more leverage than you might and can sometimes eek out more information for you.

Im not saying I agree entirely with this behavior. Specifically i think ghosting an applicant especially later in the process is unforgivable.

But I also think its your personal responsibility to figure out where you stand technically and socially. Find resources, people who care about you and ask them for feedback.

The interview process is terrible and its only gotten worse in the 25 years ive been working. Hang in there and do your best, but try and keep your expectations reasonable; you'll live longer.

I am a staff engineer with 10 years of experience who has recently gone through the hiring gauntlet and dealt with something similar.

I interviewed for a company in May that gave me a takehome that was very vague and probably would have taken my entire weekend to do it. So I declined and moved on.

My rule of thumb though is to spend no more than 2-3 hours on a takehome challenge. If you think it will take any longer, don't do it. And lack of any feedback from the company pass or fail is ridiculous to me.

As for your submission it LGTM. I'd bring you in and probably ask about how you'd want to improve it if you had more time, expecting the usual best practices and scalability things. I'm not sure your Python experience, but given the context of the problem space, I'd ask why you wouldn't consider other approaches like adding the url metric logic in a decorator or some other modularization that could be reused to "performance test" future API calls. I'd ask about your submission's lack of tests (getting at unit tests here) and how you could refactor your solution to facilitate better test coverage if you had more time. Deepdive about the tradeoffs of using Python coroutines as opposed to other concurrency methods.

Unfortunately there's always going to be a desperate developer out there who will spend every waking second overengineering a solution that will set a high standard for everyone else to be compared against.

Wishing you the best of luck!

>I'd ask about your submission's lack of tests (getting at unit tests here)

Every commenter hit on this. I have my reasons. But it's so common that next time I'll be for sure doing it.

Lesson for me and everybody: for take homes write tests.

>I'd ask why you wouldn't consider other approaches like adding the url metric logic in a decorator or some other modularization that could be reused to "performance test" future API calls.

I would consider it. I just didn't do it lol. You can already use that test route to run integration tests, but the logic in that route is easily placed in its own undecorated function for usage outside of flask .

>Unfortunately there's always going to be a desperate developer out there who will spend every waking second overengineering a solution that will set a high standard for everyone else to be compared against.

I actually could've spared the time for more hours on this. But they specifically requested I spend no more than 4 hours on this. I think they've gotten things that obviously took much longer then 4 hours and they didn't like it. I think maybe I should've done it anyway just for adding unit tests. They can maybe tell that something took more then 4 hours if a project took 16 hours but likely not one that took maybe 6 to 8.

Looking at the requirements, they feel very understandable to me since I have worked with Flask quite a bit. Then when I look at your repo, I see poetry and quart, both things which aren't even in the requirements. If I had written these requirements, and received your assignments, I would have passed on them the moment I see something I don't know or didn't specify (like quart).

I don't know about quart but what is wrong with having poetry. It shows that you have created a reproducible way to get the required libraries.

There is nothing wrong with it. I mean there are probably a few dozen others things he could have used. But if my requirements did not mention quant and poetry, while I know that I could get this to work with only docker-compose, I’d fail this take home and move to another applicant who followed the requirements and carried out the process I’m familiar with. Yes sure… he sure he shows that he knows that other tools as well… cool, but what if this was a well defined sprint ticket instead and he came back being off requirements, not matching up with the process? I’m sure there must have been other applications who didn’t diverge at all… delivered me only a working docker-compose, perfect. I’d prefer moving forward with those instead.

There's some nitpicky bits (like how "utils.py contains two utility functions" is then followed by _three_ bullet points :P ) but one big one that jumped out was using "zrank" to get the new score.

That's not going to return the right value most of the time - for instance, in the example from the docs at https://redis.io/docs/data-types/sorted-sets/ has this:

    zrank hackers "Anita Borg" is 4

but "Anita Borg"'s score is 1949. The operation you likely wanted was "zscore".

It would be straightforward to check this, either manually or in an automated test - simulate hitting "/api/abc/" once and "/api/xyz/" three times. The result of 3 of the 4 requests will be wrong:

    for /api/abc/: will have count=0 because /api/abc/ is the only entry in the sorted set
    first one for /api/xyz/ will have count=1 because /api/abc/ sorts before /api/xyz/ and they both have a score of 1
    second one for /api/xyz/ will have count=1 because /api/xyz/ has a higher score than /api/abc/
    third one for /api/xyz/ will have count=1 because that's still true

Now, it's true that the assignment didn't say anything about what the service should return: you might have done better on this assignment by returning HTTP 204 and an empty body.

IMO when a "super senior / staff engineer" chooses to write code, they should write WORKING CODE and verify that that's true.

>IMO when a "super senior / staff engineer" chooses to write code, they should write WORKING CODE and verify that that's true.

You can spin the code up with "docker-compose up --build"

You and everyone else can then verify your statements on my project by running this:

   docker exec -it flask poetry run http localhost:5000/api/abc/
   docker exec -it flask poetry run http localhost:5000/api/xyz/
   docker exec -it flask poetry run http localhost:5000/api/xyz/
   docker exec -it flask poetry run http localhost:5000/api/xyz/

(By the way the response on these calls actually returns the current count of the specific route for easier testing. You can see whether your hypothesis is real by watching the results unfold on each request.)

Then run this:

   docker exec -it flask poetry run http localhost:5000/stats/

and the results are:

   [
    {
        "count": 3,
        "path": "xyz"
    },
    {
        "count": 1,
        "path": "abc"
    }
   ]

Which is correct. Which means You are wrong sir. You did not test my code. Literally, I just ran this in my code, you can try it if you have the time.

I mean you're not the only person to make wrong assumptions on this thread. I'm getting the sense that reviewers just take a quick look and make rash judgements and then throw it into the trash. I'm betting it's not too far off from what the reviewer did for me.

You didn't run the code, which is fine, it's not expected of you. But shockingly, I think it's quite likely the evaluator at the company didn't run my code either and that is not expected or appropriate imo.

I think it's safe to say that This thread in general and your response included is becoming an accurate representation of typical shenanigans that happen for take home projects.

(Your mistake here fine, not a big deal, you're not evaluating me just giving me advice so getting things wrong is understandable in your case)

So you might have an answer as to why you got a poor evaluation. Multiple people have had trouble following code, and for good reason, what's going on here?

    # - pipeline insures that the zincrby and zrank call happen atomically and prevents race conditions.
    # - zincrby increments the score on the string, zrank fetches the new score
    pipeline = redis.pipeline()
    pipeline.zincrby(URL_PATH_COUNT_DATA_STRUCTURE, 1, path)
    pipeline.zrank(URL_PATH_COUNT_DATA_STRUCTURE, path, )

    data = await pipeline.execute()

    # returns the path and current count that was just made.
    return {"path": path, "count": int(data[0])}

So you make a pipeline so that the zincrby and zrank can be atomic. But the zrank seems to be completely pointless, it does not, as the comment suggests, "fetches the new score." The reason the counts are correct is because you are pulling the first reply out of the pipeline which is the result of zincrby, which is the new score.

So if I were evaluating your code I would wonder

  - Why is there a pipeline here at all?  

  - Why is it zranking?  

  - Did the comment go stale and the candidate just has poor attention to detail?

  - Is the comment the candidate telling me that they don't know how zrank works and they don't know how pipeline works?

  - Why is the comment lying to me?

  - Does this super senior engineer think "insure" and "ensure" are the same word?

The above code seems to my reading to be identical to

    score = await redis.zincrby(URL_PATH_COUNT_DATA_STRUCTURE, 1, path)

    return {"path": path, "count" int(score)}

Yes this is probably it. The biggest flaw in the code. I will note, only one person pointed this out in the entire thread and that's you.

If there's any raw legitimate claim to a rejection it's this.

Literally everyone else commented on a bunch of superficial things and no one hit on the extra zrank here. It's not even about not being able to follow the code. I think very few people even remember specific redis commands or even bothered to follow the logic.

>There's some nitpicky bits (like how "utils.py contains two utility functions" is then followed by _three_ bullet points :P )

There are three functions: https://github.com/anonanonme/takehome-sample/blob/master/ut...

I don't judge people for these types of mistakes either. It's all nits. Especially in an HN comment. But someone evaluating a job candidate, in my opinion, needs to step up his game and I'm thinking a lot of these people are treating it the same way you're treating it.

From your documentation: ----- Self Criticism or things I would've done if I had more time. testing: -----

That first part over there blew up your whole effort. No matter how good as an engineer you are, if you don't provide any unit tests, most often it's an automatic rejection.

Also why did you put that self-criticism section anyway? You don't owe them anything nor should they owe you as well.

As a final thought when you are doing the effort to apply to work for someone else then they can reject you for pretty much anything so don't sweat it too much.

Not hearing anything has little if anything to do with you. Sadly, most companies can't be bothered to follow up in any way. Consider it a signal as to how much a organization values communication and culture, etc.

I understand. It's disheartening. But it's not you, it's them.

Well hold on. With 4 failures in a row. It COULD be me. It's not always true that it's always them. (but I get your point and it's valid.)

That's why I posted my code. Was wondering if there's something fundamentally wrong here.

It's hard to conclude this. I had the same but applied to shitty companies (startups) who did not have much money and had over the top requirements.

Four? Not at all. You don't know what else they're looking for. You don't know who else has applied. Etc.

I think you underestimate how many comms-poor companies there are, how many shite hiring managers and processes, etc.

I've aced take homes and still been turned down for various nitpicks. A lot of times I think the person reviewing the code is looking for their pet preferences to be met rather than evaluating the potential of the candidate.

This mirrors my current feeling. Just wanted to be sure and not biased.

I will say from the perspective of the reviewer he is likely to not be aware of this. Likely he thinks his pet preferences should be standard practice as do most programmers think of their own.

If I was evaluating your code for production standard, then no testing is automatic failure. I assume you didn't run python formatter like black. Some of the codes aren't my personal style, but I won't deduct any points for that.

Take-home exam is bullshit in my opinion. I've done total of 3 in my career, and nothing came out of it. Zero feedback other than you are not what we are looking for. I've long since declined about any take home exams. I rather leetcode than take home because leetcode can scale, while take home is just unpaid work.

I ran the default formatter in pycharm. Oh man someone who grades on formatting is a nightmare.

I addressed the unit testing thing in other comments. First I'm not an adherent to unit testing much of the code is io based and very little logic can actually be unit tested.

But everyone, literally everyone is calling out the lack of tests. So likely I will do it next time.

> ...came back as failures with zero feedback given.

Maybe a dumb question, but did you follow up and ask for feedback? Sometimes people just don't think to provide feedback, but they would if prompted to.

Other than that, I think the code is completely fine. Stylistically I can see people having non-consequential nit-picks based on what they like but I think it wouldn't be an issue to get weeded out by.

I would email the recruiter back asking for feedback on the project. The worst they can say is no, and at best you get some insight into their reasoning.

Code review: For a senior dev, I would expect tests even if I didn't ask for them explicitly. I'm also surprised to see that the functions are in a generic utils instead of better named files.

The reality: 1. other people are spending more time on these takehomes. 2. Being more senior means that there are higher expectations. 3. The takehomes are typically a qualitative assessment

> I would email the recruiter back asking for feedback on the project. The worst they can say is no, and at best you get some insight into their reasoning.

Good tip. They already said in the rejection letter no feedback will be given.

> For a senior dev, I would expect tests even if I didn't ask for them explicitly.

Several people mentioned this. You're definitely onto something here.

The thing here is that the code is mostly IO, not amenable to unit testing. For most web applications that has a lot of logic moves from database to web app to client to route, not much can be unit tested.

Typically the best pattern to follow here is to segregate as much logic away from IO as you can and move it into pure functions. That's what utils.py is. But you'll see utils.py is so small there's not much there to be worth testing. (Dependency Injection and mocks is a more popular pattern for unit testable code but it makes things much more complicated and you end up writing a lot of mock code and the unit tests end up testing mocks more then the actual logic itself)

The fact that utils.py is so small is an indicator that this web app is mostly an IO app and you have logic on the database dependent on logic on the server and vice versa which takes a lot of time to write "integration tests" for. Integreation tests are excessive imo for a takehome project.

Believe it or not excessive adherence to unit tests is actually a quality of junior engineer. They aren't knowledgeable enough to practice nuance and to see when unit tests become completely pointless.

But this is probably just my opinion. I think it's quite likely a lot of (senior) reviewers are following your train of thought.

Honestly, your code looks great. Yes, there are thousands things that can be done better, so there will always be people saying so (just like in this thread). But that's no reason to deny a candidate a chance. Mature companies realize this and aren't really looking for the perfect code. They are more interested in the thought process behind the assignment. I think you're just having a hard luck, specially with a tight tech job market.

Impressive for only 4h of work and good caveats list in the documentation. I can't see the restriction on number of path segments in the code ("/api/ followed by 1 to 6 path segments") and they might have insisted on tests while not mentioning tests. Nobody can tell what "production readiness, and code structure" might mean for them.

>Impressive for only 4h of work and good caveats list in the documentation.

Thanks! They requested me to spend only 4 hours on it.

> I can't see the restriction on number of path segments in the code ("/api/ followed by 1 to 6 path segments") and they might have insisted on tests while not mentioning tests.

the segment amount is restricted via a default value in utils.py:

   def generate_test_paths(test_path_amount: int,
                        segment_amount: int = 6, #here!
                        string_pool_amount: int = 3,
                        string_length: int = 3) -> List[str]:

The segment_amount gets inserted into this lambda:

   segments = (segment_generator(random.randint(1, segment_amount)) for _ in range(test_path_amount))

which generates a list of strings from a pool of pre-selected strings.

Perhaps this section isn't clear?

>Nobody can tell what "production readiness, and code structure" might mean for them.

yeah I hate that.

Your code is fine. Now you've got a decent sample, in future just respond to any other takehome requests with a link to this above, since it already shows you're competent and know how to reason.

No need to waste another 4 hours per company (a ridiculously large task).

Interviewing is a process that can make anyone, no matter how great they are, question themselves. Sometimes evaluators have some weird criteria that is important to them and you'll never know why!

Have take homes become a lot more common these days? I think I've done maybe 4 in my whole career.

I think so. It wasn't like this before. Another thing that's getting popular is asking questions "related" to the job.

It's also kind of stupid in my opinion because of the way they pick the question. Usually they'll find the most interesting leet code style problem that their company had to solve one time in a blue moon and make that their test problem.

With 10 yrs of experience maybe you can call some contact and remove the first round coding exam?

Nah, no contacts in this company, or the ones I applied to.

I'm not exactly the most successful socially saavy engineer with a bunch of industry connections. I'm more just a lone wolf programmer. I like to think I'm really good, but guess not?

Reply on news.ycombinator.com