In Praise of Top Down Programming | Hacker News Reader

In Praise of Top Down Programming

i-programmer.info

95 pointskiyanwang67 comments

If I'm optimizing for my brain, my strengths, and my programming style, bottom up is better. With a particular focus on nailing down the hardest or murkiest part, then the next hardest/murkiest, and so on. This phase can be ugly code if that helps ... the point is to uncover unknowns and let points with the fewest degrees of freedom become the first stakes in the ground.

When there are no particularly hard/murky parts left, it's top-down from there, and that let's you choose the approach for various portions of the solution (such as, this part will be pure functions, this part will be based around an arena allocation, etc.).

Even though I mentioned both bottom-up and top-down, the 10,000 foot summary is bottom-up, because it starts there and the most time is spent in that mode.

Whenever I get away from that and go top down too early, I'm likely to run into some area that either gets reworked, or that I compromise on and _wish_ I could justify the time to rework it.

(And also with the caveat that some things are simple enough you could do them any way you want, and those are good candidates for top-down, which also seems more comfortable for more junior devs).

UPDATE: However, I completely agree with another commentor -- start with the UI first! I don't think that contradicts bottom-up ... it's about fleshing out both the requirements and the _potential_ requirements.

I have to do the same thing. If I spend too much time on everything else, I procrastinate on the hard stuff until I've wasted too much time. If I start with the hard stuff, I'm a lot more motivated and productive. That doesn't mean I'm not often paying a lot of attention to the overall design of the code and where/how things are being called. I'll often step up and down the layers to refractor and improve the code surrounding what I'm working on, since that code often has a big effect on how I can implement my changes.

However, I'm also in a lucky position where I'm not responsible for the UI or any customer facing stuff. My users are internal services that interface with my APIs, which rarely need to be changed.

I usually champion the "let's start with the UI" state of mind in every company I go.

This has several benefits:

- it forces to confront reality of the problem and field

- you have to talk to people, and understand their needs

- you almost get your backend public API out of it for free

After that, you don't really need to do full top down, you can split in subsystems, and works on small parts of those subsystems.

It doesn't have to be a good looking or sophisticated UI. It can be a ugly, as long as it's practical and validated by the end user.

But this works only well the if the problem can be well defined and you know how to solve it.

If you need to figure it out, top down doesn't work as well.

A downside to this is that some types of stakeholders lament the apparent glacial state of progress backfilling a UI with functionality.

A tick/tock of UI/functionality keeps those people from getting depressed during demo meetings.

If you have demo meetings that matter, you likely have a designer, in that case the UI won't be ugly.

Demo meetings that matter, now that's an anti-pattern if I've ever heard one. Only UI work makes for a snazzy demo, anything else will bore stakeholders to tears (and you of course need stakeholders if the meeting matters). The backend can be completely mocked out and stakeholders wouldn't even realize. All demos do is bely progress and make milestones seem closer or farther away than reality. The only "demo" that matters is the actual product launch.

demo meeting matters for several things:

- reassuring the people that pay the bill

- get PR feedback

- give a concrete objective with a deadline to the team

But like a lot of things, it's not useful for what most people think it's useful.

All of those things can be done through other mechanisms without distracting engineers with a dog and pony show.

I have always had great success when systems are designed from the UI first. It really helps you break a new business case down to the point where the backend code just starts writing itself.

I suspect you are coming from a design background because people who have never built a UI before are always very suspicious of this suggestion.

I don't come from a design background but my father drilled into my skull that nobody cared about my beautiful skills at work. They cared about their problems and if I could solve them.

Coding just happen to be a mean to solve their problems, but they don't understand code, and I don't understand their problem.

After a lot of frustrating client/dev interactions, I came to read "don't make me think" from Steve Krug. I realized the UI was the only language I shared with my customers to start a meaningful discussion.

This turned out to work well, and got me paid, so I sticked to it.

I coach Product Owners the same way when trying to create a backlog.

I know it's passe, but I generally like to start with a set of requirements first. In the old days, when we were doing XP instead of agile, we didn't feel the pressure to "code something, anything." Now, unless I check in a couple thousand lines of code per day people think I'm slacking.

But... not trying to say "build the UI first" is a bad choice if it works for you. I do mostly backend work on "high trust" transactions, so I tend to look at the API first and see if it matches the requirements i have at the time.

A good many times the customer doesn't spot important issues until they see it in a UI, preferably interactive. The design process is done best with a combination of formal written requirements and mockups/demos.

For some applications, like an API-driven backend made to serve other applications, you can consider that the API itself really is your UI. It's how your users interface to your code. Your users just happen to be other developers whose code had it's own UI too, maybe a text or graphical one or maybe another API.

Good point. I was interpreting "UI" as "GUI".

Exactly. A web api, a CLI... It doesn't have to be graphical.

XP as I remember it was about implementing one customer-relevant feature at a time. This isn’t too-down or bottom-up, it’s doing a thin vertical slice. There shouldn’t be code that isn’t used or buttons that don’t work.

We did work from a designer’s mockup, though.

I don't think that's XP.

Trouble is, how do you get this set of requirements?

The UI is the answer to that.

The UI [GUI, API, CLI, etc] is a statement in the "solution domain." By "requirements," I mean a problem statement.

"A form with three text inputs labeled street, state and zip" is solution while "the system should allow the user to specify an address" is a problem statement. (And ideally it should be more detailed with respect to its context: "The system should allow a user to specify an address associated with a delivery record, associated with the user's id.")

The problem is generally poorly perceived and expressed by the client, and badly understood by the dev.

So the UI is the best road to build one in collaboration with the client.

This is how I start with libraries. How do I want to use it, and start with some code that will eventually compile. It acts as an integration test too. But this starts to expose issues earlier with my understanding. Then if the interface doesn't work, it's easier to fix earlier.

"Everything should be built top-down, except for the first time" (from Perlisms, 1970s or so).

Approaches to neatly building a well-understood thing are radically different from approaches to building / tweaking / rebuilding a new, so far untried thing.

Software engineering is most valuable around the latter kind of thing; the former kind of thing is usually already built and open-sourced.

How often have you gotten to build things for the second time?

The dilemma there is that most of us only get to build things for the second time if we are working on a copy-cat product from a competitor, and those are not the sorts of projects we glorify. Everyone wants to work on new ideas, not copy-cats.

Perl had a "write one to throw away" philosophy that you should write a quick and dirty prototype to understand the business problem and then you could design the full thing with an actual understanding for how things work. Stuff was still complex back then, but probably a lot simpler. I doubt it's possible for a project past a certain size.

A lot of contract and some corporate work is about building things n-th time. A web form like many other web forms, a CI pipeline like many other CI pipelines, etc. Nothing fancy; 99% like last time elsewhere, just reshuffled to match the local conditions.

It's not exactly the same thing, but the amount of well-known stuff in it is pretty large. Here's where frameworks like Rails, or Django, or Next.js shine: they have much of the expected, n-th time code factored out, you need to glue predefined parts in predefined ways, and get an expected result.

I have the feeling that TDD-oriented people like me tend to do a lot of top-down, at least mentally. I'll frequently write the general behavior (in my case either the API or runtime params and inputs) then work my way down to make it work.

Somewhere in the middle, however, we all need to go down the layers to either store something in a database or figure out how to work on a collection of items. This turns the process bottom-up to some extend.

Then the very last phase is going back and forth until both solutions meet in the middle.

It's a constant problem with trying to talk about process. It's much easier to see the struggles of others, and not your own. I've had a couple bosses who probably could write a book on management, but to a man, their thesis about what makes them successful and which bits are necessary always misses some critical piece, usually lamented in private amongst the senior people.

One of these, talking about his own mentor, asked me once, rhetorically, "is he successful because of his theories, or because he uses the same people on every project?" Wish that some of my previous bosses were that aware. They didn't always see the detailed work their senior people were doing to keep the wheels on. You always assume they see that about you when it comes to who they chose to report, but to hear them enumerate the virtues of the team, and omit such things, that stings.

There are certain fictions we engage in different strategies for solving problems as well, and there's always a little Socratic method in the middle where we pretend like we haven't seen the solution at the end, and go through a farce of arriving at it step by step. That may reduce the importance of sudden epiphanies, but it does not remove it.

> at least mentally.

Even without TDD, that's kind-of how I go: I'll have the top-down general structure in mind, but not in the code, and write the actual code bottom-up. That way I encounter problems with the implementation details sooner, and can more readily adjust the overall structure without feeling bogged down by what's already written.

In the broader structure though it's more like a through-line: One nice input/output, written in that bottom-up-with-top-down-in-mind way, all the way through a use case, then expand sideways with the other inputs/outputs built on the same lower-level functionality.

Pure Top down only works for small problems or problems you already know how to solve. Its main issue is that you don't know whether the solution is correct until you reach the bottom.

Structured Design, which is a quasi top down method, works better in real life. Its main issue is that it didn't survive the OOP hype.

" Object-oriented design isn't top-down, even when it pretends to be. The decomposition provided by objects is a model of the real world, which is after all where the idea originated"

I've seen plenty of code in "OO" languages that is just the sort of top-down imperative function type thing being described in the article. That is, people using OO solely to encapsulate state and variable scope, using methods more like functions, versus trying to model the real world as objects. I suppose though, that's not "OOD" then?

The most usual patterns I have seen r using interfaces instead of inheritances and tagged union, and that's pretty much about it

Isn't true OOP about Messaging and not about modeling objects of the real world?

Like many other religions, OOP has split into several competing factions, each with a slightly different interpretation. To complicate the situation even more, in many areas of the world OOP has been mixed thoroughly with the local pre-OOP beliefs from the dark ages.

True OOP believers must necessarily code in Eiffel. /s

Bertrand Meyer's book Object-Oriented Software Construction is just about as good as it gets for defining and justifying OOP. Still worth a read today. That said, functional programming and hexagonal architecture are probably more useful.

I found a lot of holes in the justification. The scenarios make a lot of questionable assumptions about how the future will change. I'll see if I can find a certain critique that used to be floating around the web...

I don't think I would trust any system that focuses purely on modelling the real world.

We often have to model legal constructs. Short of working for organized crime there is really no way around it.

The thing they don't tell you - or at least didn't tell me - at school is that the "real world" isn't... the real world. Your Cow and Car objects aren't supposed to be modeled after the real thing. The terms "cow" and "car" as you understand them aren't the real thing either. They are labels, part of some classifications, some taxonomy. Those are all human conceptual constructs, with their own synthetic rules that may or may not have some predictive power about the real world. In this sense, I'd say legal construct are much better representation of what we're modelling with OOP, because the law is obviously a human-invented game with human-invented rules. Cows and cars? They're that too, but it's not obvious.

This is basically the approach I learned from HTDP, and it's helped me greatly for tackling some tougher problems. Basically you make a "wish list of" the functions/steps you'd like to be able to do, and then as you work on writing those functions in your wish list, you do the same thing again, going down and down until you hit the bottom and do concrete action with primitives or whatever. It's really cool to see how this can work by composing pure functions and types.

https://htdp.org

There is a third way: define, explore and build a model of the problem space your program is supposed to be addressing. This forces you to understand its properties much better. Once you understand it better you can likely implement in any of the common programming languages. This is also why I like languages like Scheme or APL or k for exploration.

Often the UI addresses only a subset of this space -- it is only the visible part of the iceberg as it were! There is a lot going on under the UI surface. Not to mention you may want more than one (kind of) UI.

Sounds interesting. Can you give or link to an example?

The idea is quite old -- See for instance https://en.wikipedia.org/wiki/Formal_methods

For an example see https://en.wikipedia.org/wiki/Vienna_Development_Method#Bank...

The idea is to model the key aspects as mathematically rigorously as possible while ignoring lower level details. But even if you are not as rigorous, the process of creating such a model can help clarify things. As an example, if you are developing a file system you can abstractly define operations on it (read, write, create file, dir, etc.) and one can come up with a huge number of implementations satisfying this but as you add other requirements or constraints, quite a few choices are no longer meaningful. For example you want to implement a FS that takes advantage of NVMe SSDs, they put certain constraints. So then you need a model for an SSD (erase blocks are multiples of data blocks, write must be preceded by erase, etc. etc.). The idea is to model all this using a few abstract data types and state. You need not strive to programmatically derive an implementation from such a spec for it to be useful.

For the "play chess" example a state machine is the perfect place to begin modeling. Games, as a rule, are event-driven and are commonly coupled to a event-loop that triggers the transitions of the state machine. I think top-down decomposition has it's place, but the examples used to describe it should be given a little more thought.

I think the advantage of chess for something other than top down programming is that you're pretty much 100% what the API you want for the chess state machine is before you start. But how common is this?

"Stepwise refinement" is an intuitive and powerful concept, in part because it helps you grok and debug in a "fractal" kind of way, level by level. But it does have limits at a larger scale. (Event-driven architecture is often better for big.)

It's something Functional Programming either lacks or doesn't do as smoothly. The intermediate results and state are highly useful for debugging, and Functional doesn't "like" intermediate state. Yes, it can be emulated, but the emulation is rarely good as the real thing. For one, the emulated state lacks useful variable and token names, because it's machine-generated.

Functional doesn’t like state. That’s the whole point. Functions that can be run in isolation with well defined input is much easier to debug.

In theory. Practice has proven messier to many. Maybe some have a "functional mind", but they shouldn't extrapolate their head to others. Functional has been around for 60+ years. If it were the magical productivity and Lego-modularity golden hammer, it would have mainstreamed already. (Long LINQ is also hard to debug for many of us.)

It is becoming mainstream. Not by pure functional languages taking over, but by developers adopting a functional style and by better and better support for functional programming in mainstream languages.

I'm not sure I'd agree. A lot of the use-cases are for niches or systems programming. I don't see it for business CRUD that often unless either the framework forces it on one, or they are using LINQ, which as I pointed out, can be tricky to debug if long. SQL is largely functional, but it's also hard to debug. The addition of the WITH clause to SQL greatly helped, as one can paste sections to test. Still, not as x-ray-able as imperative. Imperative still wins most x-ray contests. Maybe fast code readers don't need debugging as often? Could be, but I'm average at it.

> The intermediate results and state are highly useful for debugging, and Functional doesn't "like" intermediate state.

s/foldr/scanl

Only sort of kidding. Can you give aconcrete example of some task you'd like intermediate state around?

Also, do you know about the validate monad?

       x = foo01();
       y = foo02(x);
       z = foo03(x, y);

Contrast with:

       foo03(foo01(), foo02(foo01()) );

x, y, and z were named by humans, giving them domain meaning, whereas a debugger would give the second one no name or machine-made names for intermediate state. (Here, z, y, and z have no meaning because it's a foo-bar example, but in practice they'd have better names.)

The first is also often easier to read than the second, at least to my eyes. I know devs who can read the second quickly, but I don't think it's the norm. The LINQ "dot style" can be a little easier to read, but has similar problems.

> Here, z, y, and z have no meaning because it's a foo-bar example, but in practice they'd have better names

As would foo0x.

I don't think many functional programmers would object to writing code as you did in your first example if you think it improves readability and if you don't change what the names refer to.

Your first example is just as functional as your second. And, quite possibly, better style.

But your overall point is fair. A function, once composed, curried or having captured data, is very much like an object. Indeed its representation in memory is likely very similar to an object (mostly functional languages) or actually identical (Java). Your debugger won’t show the data in such a function but it’s happy to show the data in an object. That’s a big advantage of objects over functions. Hopefully debuggers will get better at this eventually.

This example:

    x = foo01();
    y = foo02(x);
    z = foo03(x, y);

could easily be written in Haskell as:

    let x = foo01
        y = foo02 x
        z = foo03 x y
    in z

My thinking is that functional languages don't preclude you from having intermediate state, but I will agree that composition is used often.

Typically when working in a repl I don't really miss the intermediate state because my debugging consists of:

The lambdas must flow. λ> foo01 "expected result" λ> foo02 "UNexpected result"

Then I'll debug foo02 in isolation.

> foo03(foo01(), foo02(foo01()) );

I think I'd write this as the let example above with variables. Otherwise I guess I'd need to use something like:

    foo03 foo01 . foo02 $ foo01

Here's a Haskell playground link with these examples if you're curious or had something else in mind and want to modify the example: https://play.haskell.org/saved/MeRGyCjr

I'd go with this, which I find the easiest to read:

  x = foo01();
  foo03(x, foo02(x));

In my head I tend to see code as a directed graph, and this one most closely matches that graph without creating too big a jumble on one line. The first one that also has y and z has additional nodes and edges, making the graph more complicated than it needs to be.

Your second one would have the simplest graph, except on one line like that it obscures that foo01() is called twice. If that second one was actually foo04(), then I'd prefer this as the simplest form (which I'd actually originally written before noticing foo01() was in there twice), because it has the simplest directed graph of all, and the code matches it very closely:

  foo03(
      foo01(),
      foo02(foo04())
  );

If the variables and function names were long enough, then I'd use that last format even with my first example, to avoid the whole "jumble of letters and numbers" while still keeping the directed graph simple:

  the_big_x = the_big_foo01()
  zimzamfoo03(
      the_big_x,
      bang_bar02(the_big_x)
  );

In my more complicated personal projects, I'll usually do this in comments in the code. Basically pseudocode to just expand out and then I'll either write things in between the comments that actually add the functionality or I'll just leave it at the top as a kind of "this is what this iteration does" because I'll also frequently complete an idea and then I'll have some other idea with it that requires refactoring and I'll make my second file etc!

Top-down programming works great when using GPT-4 as a coding assistant, as long as it is told to not stub functions, use placeholder or TODO comments, but use as many undefined functions as it wants. But when it starts calling the same function recursively, it's time to tell it to stop the top-down approach and write the functions that does the actual work.

It's not all one or the other, I'm more of a layer-at-a-time developer and within a layer development can be inside-out (start with an algorithm and add an interface) or outside-in (start with an interface and mockup, then add the internals later).

I mostly like to choose a data structure that's handy for modeling the data I'm worried about first. Then I write code to manipulate that data, including getting it into and out of other formats if necessary. Then I worry about what the application really needs to do with it. Once everything's a presentation or transformation of the central data structures, the code becomes fairly obvious. One I have the data handling and the UI, often the UI is just another case of presenting or updating the data.

Meh. Just Common Lisp. Have your cake and eat it from the top or the bottom, at the same time?! mmmm, maybe a new paradigm is in order ;)

Forth will echo your "Meh". How much of top-down is because bottom-up is so @#$%! hard in non-interactive languages.

In a non-interactive language, you need to create a "harness" (aka, a program) to run your lower level functions (here I mean in the "top-down" sense of functions at the bottom of the call tree) independently. A test framework could accomplish this, and so you'll see people who do a more TDD-inspired approach will have an easier time conceiving of how to develop their program from the bottom-up. If you don't have a test framework or other framework assisting you then you end up writing "throwaway" programs (I often don't throw these away, they become utility programs when I'm done, though more for diagnostics and testing than for end users). People see that as wasteful, so they don't do it even if it might speed things up for them because it gives them earlier validation of their design and requires less rewriting if they find their design is invalid.

Interactive systems like CL and Forth (as it's commonly implemented) give you that framework for free, it's already written. So why not take advantage of it?

In a top-down vs bottom-up world, the Mikado Method is the middle-out approach.

Thanks! Will read more carefully, but at a glance it sounds suspiciously similar to effect sketches and feature sketches from M. Feathers' book on working with legacy code, rather than a design methodology method per se. (is that the case?)

EDIT: oh I see, it's basically the same concept as effect and feature sketches, but at the design stage rather than at the "dealing with existing code" stage. The intent is to figure upstream and downstream dependencies in your design in a structured manner. Cool.

Frameworks often impose a kind of top down structure.

That site's layout is like 80% ads, 20% content!

Reply on news.ycombinator.com