Ask HN: What is it like working on a large dynamically-typed codebase? | Hacker News Reader

Ask HN: What is it like working on a large dynamically-typed codebase?

21 pointscrummy55 comments

I have worked on small-to-medium projects but only in strongly typed languages such as Java or TypeScript. I rely heavily on my IDE to tell me where a field is used, or to allow me to rename a method and still be confident that the callers have been updated, or add a parameter to a method and know my code won't compile until I've fixed the callers.

I can't see how this can work at scale in a large project without compile-time typesafety. I suppose 100% test coverage could provide some of the benefits, but not all.

If you've worked on a large Python or JS codebase, what was it like to work in? How did you make small refactors for example?

View on HN

I used to work in very large Ruby codebases, and have also worked in some midsized Python ones. If you rely heavily on your IDE, these languages will be painful. Rubymine is surprisingly good for what it is, but it's solving an intractable problem.

The mindset is a little different in these languages and codebases IMHO - rather than a "very scientific" approach to refactoring and programming with a heavy reliance on the IDE, compiler, and analysis tools to get things just "right" on the first try, there is a more "system in the loop" approach to refactoring. Code is refactored, tests are run, failures are addressed using error output, and then full-stack or end-to-end tests are performed to validate a fix.

You're still doing most of the same thing, just at a different point in the development cycle. I think it's hard to argue that refactoring isn't "more expensive" this way. On the flip side, there's no "it compiled so I ship it" going on - an end to end understanding of the refactor and what it touches is essential.

Also, oftentimes your codebase ends up brittle enough that you need to start introducing speculative runtime / "testing in production" features - feature flags, phased rollout, in-depth error telemetry, etc. This is again expensive but at the end of the day I suppose a brittle core can lead to a more resilient system overall.

This question is bait for HN. Whether it's a huge pain or not (I happen to think it is) the "dont use a dynamic language for large projects" crowd here is very vocal and they will certainly tell you it's an awful experience you should avoid.

Yea, but you don’t always get to choose. Or rather, maybe it’s not that bad all things considered. And either way, what are the takeaways?

Worked on a 1M+ LOC python codebase (it was roughly a precurser to kubernetes, but built around chroot and included such niceties such as code deployment, database management, etc.). It was fine. I now work on a 158k+ codebase in Java, and it's also fine.

Refactoring in Python wasn't automatic, but it wasn't hard either. Most folks weren't clever in their work, which is to say there wasn't a lot of non-obvious metaprogramming going on. It meant refactoring wasn't a hard task most of the time.

There's a lot of things I miss about Python. The lack of insistence that everything is a class, the lack of DI bullshit, a lot more functional programming tools... etc. And yes, I'd rather monkey patch than deal with spring's DI fuckery. Every. Day. Of. The. Week.

Your post reads like Java is the only statically typed language

The past 5 companies I’ve worked for have used Python, Java, Kotlin, and a home made language. So, while Java is not the only statically typed language, it is one of, if not the most prevalent one.

Go and Rust are becoming quite popular, and don't have the DI issues you complain about.

> allow me to rename a method

In Ruby we generally rename a method but alias the old method name to the new one. Flat out removing a method (whether by renaming or deleting) is nearly unheard of, especially in a legacy codebase.

I don't rely on my IDE much at all. I use runtime introspection to discover the objects at play and what they're capable of. If I'm trying to figure out where a method comes from and it's really unclear, I can access the class, call the `method` method with that method name, and get the source location of that method's code.

There are lots of other techniques and approaches to working with dynamically-typed languages. I'm mainly familiar with Ruby so that's why I'm talking about it.

The key takeaway IMHO isn't that dynamically-typed languages are "missing" features from statically-typed languages, therefore they're somehow "worse". It's that many of the techniques concerning programming in general are simply different. Whether or not that's better or worse in the long run is really subjective by and large. (Personally, I don't like statically-typed languages. I really don't.)

There are many messages describing the pain of dynamic languages, but my subjective impression is that a lot of it is about poor quality code, rather than lack of static safety.

Case in point: I wrote and maintain a mid-sized critical service at work, say 50k LOC or so. Yes, testing is an issue, you do find it's hard to test all branches etc, something statically typed would be a bit easier. But it's fine actually, well tested, well thought through, and refractors are fine.

Meanwhile, my python service connects to a C++ service written by a contractor, which is absolutely awful and crashed all the time, or worse, produces totally incorrect results. No doubt it compiles though.

One thing which is real is that barrier to entry to dynamically-typed languages is much lower and there is simply more crappy code around. Also these tend to be used by people who are programmers but not software engineers. Data scientist don't write prototypes in C++.

If you have a disciplined, experienced team; good documentation; good coverage with automated tests; good linting; good code organization; the team avoids clever coding (e.g. lots of meta-programming), and you're comfortable leaning on UNIX as a larger part of your IDE, then it's not too bad. Of course, everything is a tradeoff. You may find that some things with dynamic typing become easier, or possible that weren't before.

It doesn't work well, not for everyone on the team at least. Static typing helps alleviate a lot of the bugs you'd normally get with a dynamic language. As of my opinion now after working on a medium/large size dynamic js project. It bit us more than I can count and simply adding in ts helped find a few long outstanding bugs that kind of went away as the develops found those issues during the migration.

There will always be opinions on benefits of vanilla js, but anything that needs to scale (team wise and communication tax) is simply easier with a static language.

I worked in a large python codebase with no types and no tests.

It was.... barely ok. Honestly I found that the biggest problem with no types was less specifically about "oops you passed a dict in but it was expecting a string haha runtime error" and more about:

[line 2359 of somefile.py, 20 calls deep]: def snafucate(id, f, connection):

[me]: ok connection is the same db connection we've been using. id is the UUID. What is f though? (clicks find-usages and picks the probably correct call site)

[that call site]: def snafucate_caller(f, connection):

[me]: [repeats this process until I deduce that f is at various points in the codebase at various call sites and sibling methods: a file IO object, a filename str, a custom type that represents a file path, a 2-tuple of (filename str, file IO object), and in one code path that I think it's actually unused, a dict that's never populated with anything.

And just lots of crap like this. It's some awful combination of poor rigor about naming, not consistently using the same schemes for args (do you usually take in IO objects, strings representing paths, or t.Union[IO,str]?), no testing, poor documentation, no type hinting, and of course the good ole' "this code has had 12 owners across 20 years, ranging from well-meaning new grads, to "experienced" people who are bad at coding, to ninjas who are too terribly clever for the rest of us to hope to keep up".

And yet somehow we still got stuff done in it. And one of the things I tried to do when working in it was either adding type hints once I figured them out, or if they were too arcane (like Union[Dict[Any,Any],str,IO,Tuple[str, IO]]) I would try to squash them down enough to make the type sane. As I was slowly dragged through the codebase, the types slowly started peppering in, and it got easier and easier, especially as I got promoted and had some juniors under my wing who I could ask to help do the same gradual-typing-as-you-spelunk process.

I kind of miss it tbh, one of the things I was doing right before I left was toying with proposing an official set of domain-specific type hints and type aliases that would've made the whole codebase a lot easier to follow if people were willing to join me in my quest.

>if they were too arcane (like Union[Dict[Any,Any],str,IO,Tuple[str, IO]])

How about

    myweirdtype =Union[Dict[Any,Any],str,IO,Tuple[str, IO]]

Than just reference it by name.

It's pretty terrible IMO.

* IDE refactoring is impossible because it can't infer the types, so if you do want to make large refactors it's a huge time sink

* It takes 3x longer to read and understand what the code is doing

* Requires almost twice as many unit tests for things that would normally be resolved by the compiler

* ci/cd can actually be slower since there's no incremental compilation

And please note: whilst Python tooling is brilliant, absolutely brilliant, it cannot match what you have in eclipse for java tooling or msvc++ for C++ or C#.

And pytypes doesn't really change that.

While this is all true, some things are much easier in Python because it is a fairly high level language. We had an in-house language for web pages (history) whose runtime we had upgraded to Java (originally, C++). Trying to write tools that analyzed and helped debug those pages was painful, because... everything had to be reified into types, and what Graham says about sculpting is true. Java is an awful language for exploratory programming.

Horses for courses.

You still need types for objects in Python - the only difference is they're in your head (or annotated)

Yes, but their implementation can be sculpted without as much protocol (interfaces et cetera).

And in most places exploratory code becomes production code because that’s life.

Sometimes. And that's fine, for values of "production" that are "occasionally we need to run this to understand/debug this other code".

And if you think Eclipse tooling is great, wait till you try IntelliJ!

I don't think the codebases I have worked on can be considered large as such. However, to me, they were actually easier to change/refactor. There was less boilerplate, and more concrete implementation than what I see in static language projects. I usually have a harder time working with static languages because of so much boilerplate they usually have. I seem to be in a minority though.

Currently working on a 7-8yr old Python monolith with 500k LOC and overall it’s been far less painful than it would be if it hadn’t been for the gradual adoption of typing and strict mypy configuration. The introduction of types has supercharged PyCharm which is already rather powerful at inference so refactoring and exploration has been rather easy. The villain of the story here was not dynamic typing as much as it was the usage of Python dictionaries as a data interchange format where no amount of typing or IDE smartness can untangle that mess. We’ve been slowly replacing dictionaries with models, eg pydantic, which brings a massive quality of life improvement but it’s been arduous and error-prone. All in all Python gives you all the tools needed to create a coherent codebase regardless of size but that requires disciplined engineering and a commitment to incremental improvements.

Agree very much with this. I work with 50k per repository and around 10 repositories for the last 4 years. Strict mypy rules do the trick, linting flake, pylint do wonders to the quality, no re-using the same variable for different things, no generic types in the interfaces (dicts, pandas dataframe).. There's no additional pain compared to our go code bases except not accounting for the compiling part.

Thank you for that, it’s encouraging to hear others can get statically-typed-like quality of life following sound practices

I've worked in multiple large dynamic and static codebases. Everything is a major PITA when you have a large codebase. At least that's my experience thus far.

It makes you really good with grep / regexp, as you can hardly rely on IDE navigation, refactoring, call hierarchy, etc.

The problems with both stem from things like lack of (behavioral) tests, and lack of cohesive architectural principles (SOLID - what's that?). There were always "good reasons" for those problems being introduced and kept. Sometimes, there were attempts to "fix" the bad patterns that were root causes of bugs, but they were likely short-lived attempts before the team had to focus on shipping features again.

Static typing often means people skip tests and that can lead to an untested, incomprehensible system that correctly passes types.

Also, strong typing and dynamic typing are not opposites.

https://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic...

Biggest problem I had with a largish Python codebase was code navigation and knowing the structure of the data going into and out of methods/functions. I had to attach debuggers just to see what was going on more often than I think should be necessary in a professional setting.

I've worked on several large ruby codebases. It's fine. Unit test coverage has always been pretty good.

As much as I like typescript I've not really desired types in ruby. For small refactors you tend to be changing just one small part of a class hierarchy or interface and you're not going all over the place updating class signatures. For radical changes you're replacing an old concept with new in every location so you'd be giving it a bit more eyeball.

I'd also point out that people didn't have IDEs like that in the past even with strongly typed languages yet they still managed. I definitely use that tool now, but I see it as only an incremental improvement, you still have to know your codebase and understand the change you're making.

To address the meta question of whether you "should" do this or not I think it's worth considering the world when these choices were made. The choice used to be (essentially) ruby/python vs. c++/java. C++ and Java absolutely suck (imo) to write all day. Whereas python and ruby are more of a joy. So you get to hate your environment and language all day for the eventual upsides of having a large codebase, or you get to enjoy your day to day with a big problem waiting for you if you hit the success metric of having a large unwieldy codebase. Most startups it's so obvious that you'd choose the first one. Then you can hire the masses to fix it later!

Maybe with languages like Go we can have both now?

In my experience, it's pretty difficult, particularly in middleware code that passes values received from one part of the code to another. There was a lot of code tracing, both up the stack - to figure out where the values are actually coming from, or down the stack - to figure out what the code that ultimately uses them is expecting. I ended up leaning on interactive debugger sessions pretty heavily, since you can peek at what kinds of value are actually going through the system, but that tends to emphasis the "happy path cases" and it can be harder to track down the edge cases.

> How did you make small refactors for example?

Easy, you don't. All code is write only, and if something does need to change, you simply insist on a rewrite and creating an even bigger mess than what existed before.

I’ve worked on a 700k+ LOC Perl codebase. Testing and linting made it possible. I’m pretty sure that a similar amount of tech debt would have been there in another form if it has been a typed language.

In Ruby? It's hell! What are those parameters? Which methods does it support? Is it nil?

In Clojure? A breeze. Data is just data, no behaviour. Destructuring shows what a function requires right in the parameter definition. You are already using schemas for validation so you can annotate them in all the functions that receive/return them.

IMO dynamically typed OO languages scale very poorly with codebase size, unlike their statically typed counterparts.

Writing tests help with refactoring.

> If you've worked on a large Python or JS codebase, what was it like to work in?

Rife with footguns.

Things broke all the time until we eventually re-added some form of static type checking.

> How did you make small refactors for example?

We'd write tests first to validate that _wrapped_ large chunks of functionality, commit those, then refactor. The key was to write these tests even if the existing code already had "unit tests" because we often found issues this way.

I worked with large-ish Python codebases focused on “full-stack ML“ (data pipelines ending with trained models and APIs for using these models). Some of my colleagues relied on PyCharm for refactoring. I preferred to use cli tools (vim + tmux, ripgrep) and ran a linter from time to time. Ideally, we should’ve integrated the linter with CI, but it was in a deep backlog.

Pycharm seems to be great with all kinds of refactoring.

> Ideally, we should’ve integrated the linter with CI, but it was in a deep backlog.

This is a P0 for me in a Python project, crazy to backlog something so useful. You can even turn it on for only errors so it's a little quieter, then gradually address < Errors over time.

It sucks. I tried to refactor a large JS project and it almost ate itself. I converted it to TypeScript and then refactored in a few hours.

I'd add, it's more than just the language. I've just started working with Dart, which, as a language, is broadly comparable to TypeScript, but the big difference is how much more pleasureable and productive the Dart ecosystem is, vs anything else I've ever used.

I don't miss it all that much, though in general I am a fan of types. Granted, I have never worked on a "large" statically typed project. I think if the language has pattern matching that helps quite a bit.

(About 72k LOC, not sure if that's considered "large".)

EDIT: Well, I guess I did work with some Hack code while contracting at FB, so that's definitely a large code base.

Refactors in a large dynamic code base are only small if the API you're refactoring has a small footprint in usage. Others have already given reasons why: tests will break, runtime issues will crop up, etc. The lack of static typing can really hinder refactoring of any size--even of "internal only" code if metaprogramming has been employed.

Code organization can go a long way but even then you will want to be very familiar with source. Talking about a large code base, you have to be even more intimately familiar since things begin to leak even with decently-strong organization. Linters go a good way in enforcing consistent style but that's a relatively small contribution to the issue.

It's fine. Usually projects will develop conventions for handling dynamic types, it's not like you never have any idea what type things are. Places where things are ambiguous you add in tests or asserts to make sure the callers are doing what's expected.

Personally found that this really, really sucks and I’d prefer to never do it again.

If you really want to just give it a go and see, pick any part of WordPress and try to make a contribution (or simply change something locally). The editor is React/JS and the backend is legacy-ish PHP.

Source: I've worked in several "too large" codebases. C#, ASP, Ruby. I thought Visual Studio + C# + Resharper was an extremely potent combination. Despite that I prefer the Ruby ecosystem these days.

One glaring issue I often see overlooked is this: in nearly any project, most complexity and runtime errors tend to come from external data sources: databases, external APIs, user input, etc. The godlike powers of static languages and advanced IDEs don't help you so much there. So in practical reality for most projects this greatly levels the playing field between dynamic and statically languages.

    > I can't see how this can work at scale in a large 
    > project without compile-time typesafety.

Moving from C# to Ruby, I thought compile-time type safety was going to be the biggest issue. It has been extremely close to a non-issue. This is going to sound almost absurdly low-tech but the answer for me is literally just "use descriptive identifier names" which is something you should do anyway. Method/function names should strongly imply the return type, ie "get_user_ids()" and not "get_stuff()". This makes most type snafus glaringly obvious; "current_temperature_centigrade = get_user_uids()" is glaringly wrong.

   > I suppose 100% test coverage could provide some of the benefits

Yeah, that's part of the idea. The idea is that you should have close to 100% test coverage anyway and your tests are going to blow up if you screw up your types.

    > I rely heavily on my IDE to tell me where a field is used

If you have nice descriptive identifier names (see above - something you should be doing regardless of dynamic typing) then grep or silver searcher is usually far superior in speed and functionality to any IDE's built in "find in files" IME. And your grep-fu (or ag-fu) is going to be useful in all kinds of situations. When feasible, master tools that will serve you for life instead of getting good at some IDE you'll ditch in a few years.

    > or to allow me to rename a method and still be confident that 
    > the callers have been updated

This is definitely something you give up in a dynamic environment, at least the auto rename part. However, the finding part is pretty well handled (see above).

    > If you've worked on a large Python or JS codebase, what was 
    > it like to work in?

For Ruby, at least, here are my conclusions.

1. You do make some sacrifices as discussed above.

2. In exchange you get a massive gain thanks to being able to work with, prototype, and debug live code in the Ruby console. I have found this to be more than worth the sacrifices. Also in my experience people who don't like Ruby generally are people who have not taken advantage of this and therefore only experience the downsides.

3. Overly large monoliths are hell in any language, dynamic or not. The flaws of a dynamic language are exacerbated in overly large codebases, but I've seen some sheer monolith hell in static languages too.

4. Sticking to a well worn paradigm e.g. MVC alleviates a lot of pain regardless of typed vs. untyped.

I haven't used Pycharm but Rubymine for Ruby is pretty amazing with refactoring help in that regard.

I find the only way it’s workable is you write a ton of unit tests that essentially mimic a compiler.

depends: how much do you like dropping a breakpoint and then fucking around at a REPL?

Pain. You write endless tests to ensure that you cover for all types coming into critical code paths, but you can't cover for all of them. At the same time, you have no guarantee of the type of something like a function argument, so a huge amount of time is wasted if you accidentally misread or assume the wrong type.

Data scientists giving me code and not telling me what Python libraries they are using or which dataframe to use has caused a week of headache, personally.

I was just saying in another comment how I wish the ML community would replace python with a statically typed language with truly concurrent multi threading. FastAI tried to pivot to Swift but unfortunately failed.

You can write `assert`'s in your code at the top to limit the scope of types coming in or handle them if the values are bad.

A nightmare, and tests don't do as much as old boomer TDD people say they do. IDEs can still do things with dynamically typed languages, but if you have an object that calls some very generic method name like `get()`, there are probably many classes with that method. The IDE can try to guess the one but it's a lot of visual noise and less precise than just being able to infer it accurately.

Languages that support type annotations (Python 3) help and AFAIK most good Python teams working on relatively recent code bases use them.

Also, even if your IDE can pick up on function usages elsewhere, if they are invoked or referenced in a non-standard way, they will probably be missed (such as Function.apply, as arguments, etc.)

Reply on news.ycombinator.com