Ask HN: Software design patterns for projects without a scope?

I personally think that you’re doing the right thing. Thinking too much about extensibility too early or applying patterns often makes your code overengineered, and in no way can guarantee that it can be extended in the right direction.

Having to rewrite existing features differently is mostly a consequence of you not being an expert in the business application you’re writing.

Maybe if you’d invest more time upfront to understand deeply what the application is for and how people are going to use it, you’d be able to spare a few mistakes.

But even being an expert and talking to people doesn’t shield you from that problem. There is always a new feature that contradicts how the code works. It’s just a consequence of software being so malleable.

As a car driver, you can’t ask the car maker: “hey can the needle on the speedometer turn red when I go over 60?”, or “hey I need this car to work with both gas and diesel?”

This would be perfectly reasonable in a software equivalent world.

In the end, you just need to explain this to your client…

Thanks for the comment. I am generally a technique expert rather than the domain expert in a project, so your comments about "being an expert" definitely makes sense.

Try to reduce the number of lines of code, and, at the very least, use languages and tools with refactoring support.

I find that I am much better at delivering ad-hoc solutions when the codebase is small.

I do much the same kind of development.

Document the design choices you made and why.

That's it. That's your answer. If anything, spend even less time worrying about 'proper' abstractions. As you've seen, you'll wind up guessing wrong and making more work for yourself/other future maintainer as often as not. Keep it simple.

At some nebulous future endpoint, if a client reaches a growth stage where they are ready to take the next step beyond small-scale prototyping, great! It's a good problem to have. Ignore the code you've written and instead read the history of your design decisions that got you to where you are. Business workflow will have co-evolved, and some of your decisions would now flow differently if re-evaluated. Write down the changes, and there's your rough draft spec sheet for version 2.0.

I second this approach.

I often find myself in similar situations as OP's, working with a client to understand a complex or ill-defined problem space and developing solutions in an agile way.

Neither you nor the client has a (functioning) crystal ball and so all that you can do is make the best decisions that you can with the information available at the time.

I've found that documenting Architectural Decisions as Architectural Decision Records (ADRs) helps build a shared understanding and ownership of these decisions amongst all concerned. They also prevent me from becoming bogged down and second-guessing myself: Here are the available options, here's what we decided and here's why.

No need to make ADRs overly long or complex. I use this "Lightweight" template:

https://github.com/peter-evans/lightweight-architecture-deci...

That's a great answer I hadn't considered. Thank you!

You're doing it right. Consider how truly difficult the environment is you're constrained to operate.

None of the users know what they really want, because the end users don't know what's possible, nor how difficult any particular feature is, so they're guessing in the dark.

You don't have the years of experience they have at working their jobs, and knowing what is quite easy or quite hard, in the same manner as the above paragraph.

In spite of a heavily information poor environment, you and the users work together to build systems that meet their needs, and they are happy.

Code is the crystalized knowledge of the program domain gathered over time. You're correcting that as you go, rewriting as appropriate. It sounds to me like you're doing refactoring as appropriate. Using the word refactoring should help you with the search engines quite a bit.

Looking back on things, it seems obvious what the right way to do something is, because you now have far more hard won knowledge you lacked back then.

Again, don't judge yourself harshly. It sounds like you're going fine.

Sounds ok to me. Probably the most flexible architecture I worked on (back in the waterfall-ish quarterly delivery schedule days) largely consisted of a rules engine built in an Oracle DB with a JSF front end that would dynamically build the display based on the fields present on the work item. This meant we could elevate things like field changes and some business logic changes without elevating the monolith just by making Oracle changes.

It really worked pretty well. The problem was, the business got used to the easy/fast changes. Then they wanted to make some larger changes involving integrations with a bunch of other systems. To do it right we would have needed rewrite/rewrite a large part of the system. The system could have used some modernizing anyways. Nobody (in management) liked my opinion. Well, 2 years after leaving that team they hired another company to rewrite and modernize the system... I guess they took my suggestion after all.

Personally, I think our existing languages and tools are still too primitive to avoid rewriting considerable amounts of code as requirements change. At some level, rewriting is inevitable. We have developed some techniques for minimizing it. The shift from assembly to C reduced the amount of rewriting necessary when the underlying ISA changes, for example. But rewriting sections of code still seems necessary quite often, regardless of paradigm.

As far as I can tell, the best technique is to write straight-line code as much as possible, and then hoist out (ideally pure) helper functions when you notice clear duplication. Then, if and when you need to rewrite, it should be easy to understand what needs to change, and you can easily inline and then re-hoist helper functions as needed.

If you haven't read it already, I highly recommend this post about semantic compression by Casey Muratori: https://caseymuratori.com/blog_0015

The most important aspect for me is getting the strategic boundaries right; the way a company is organized might not be the most optimal, so getting deep into the business and data, and at least trying to get the high-level architecture right before starting your first line of code would be my best advice.

After that, I tend to make a distinction between inherent complexity and accidental complexity.

If it has inherent complexity, I tend to spend extra time at the whiteboard to make sure I have the best possible model based on the info I have at that point in time, and I also refactor relentlessly.

For everything else, I try to make the code easy to throw away and replace. Typically things start out simple and grow more complex after a while. Trying to retrofit a proper model usually costs more effort than a rewrite.

For me, spending over a decade deep in the domain-driven design (DDD) community has proven to be the best investment I ever made in that area. However, as DDD is getting more popular, I noticed the emergence of echo chambers and some "beginner expert" thought leaders who are starting to teach before truly understanding DDD, and others relentlessly productizing and pushing their offerings, so make sure you only engage with experts who really care. (A good heuristic to detect a beginner expert tends to be someone who uses too much DDD lingo, but YMMV)

> oftentimes I'm rewriting existing chunks of code to incorporate the new features because I chose a certain structure that no longer makes sense.

You're doing it right.

> the client usually doesn't know what comes next until they need it.

Do you know what comes next? If not then designing for requirements that may not exist is likely to take your code in the wrong direction.

Thanks for the comment.

> Do you know what comes next?

Not really. As another comment put it “hey I need this car to work with both gas and diesel?”.

What I learned from 20 years of development experience is to follow the below in that specific order.

1. YAGNI - do not try to solve a problem which is not yours

2. SOLID - try to build in an extensible way, so you can later refactor it easily

3. DRY - it is time to modularize when you start copy/pasting

So, I can say that you are on the right track.

Prediction is very difficult, especially if it's about the future. YAGNI.

It feels stupid to not have any pattern or theory or code reuse, but when there is insufficient data, the null hypothesis is the best guess.

I think: have confidence that you're doing the right thing. Write the code in the most obvious way to do what's needed - the least theory, model, pattern. That will be easy to read - as documentation for later.

Eventually, you may have seen enough of the problem (or part of it) to be confident you understand it. Then you can throw away and rewrite that part. Probably, even trying to adapt the old code is not worth it - more efficient to start fresh.

Give a bit more context on what you are already using so we are not shooting in the dark.

You are likely already using these:

With a typed language the compiler helps you refactor more efficiently.

No spaghetti code, instead move logic into coherent (ideally functional) libraries.

I’ve found the functional object-oriented style to give most flexibility for changes. (Ie. Animal.kill() doesn’t modify it but returns a new Animal object.) Copy-on-write everything, use appropriate data structures to make it efficient.

Simplicity. Build granular parts that can be assembled in various ways. This way, when there's a seismic shift in requirements, you can hopefully just rearrange some of the parts instead of starting over from scratch. Beaches aren't really affected by earthquakes.

Overarching frameworks and patterns work well when you know ahead of time what you're building; unopinionated granules work well when you don't.

Fundamentally I think you are doing it right, especially because you are not afraid to rewrite substantial parts of your code, while being pragmatic about what that entails.

Never forget: we also need to feel comfortable in a code base. our happiness matters, it's not just about agility. A tidy, continuously improving code base has more long term effects than just "shipping stuff quickly".

To dive deeper into the topic I can very much recommend two books:

A Philosophy of Software Design. Very practical, friendly book, actionable advice and some sound mental models to apply to designing interfaces, error handling, modules and so on.

Software Design for Flexibility. Spiritual successor of SICP. Very challenging, dense and in some ways radical. Typical Sussman-style of throwing challenging concepts at you in rapid succession in a logically sound and clear language. Full of exercises, questions and interesting references.

Can you give an example or two of the software design patterns you’ve found helpful in the past? I had a big comment typed out but then realized there are many kinds of, like, “scopes” in your post. Are you looking for abstractions to cut down on your time-to-prototype? To extract a “framework” from your extant software to increase your efficiency over time? Something else?

I'm happy with time-to-prototype, and wouldn't even mind spending longer on it. I'm looking for patterns that allow me to deliver the prototype, but not get in the way when adding features later in development.

Well in web development, where I work, we use web frameworks that take care of the stuff that never changes between jobs, the constants. So I guess the question is, what are the constants for you between clients? Are you starting literally from scratch every job?

edit: What the other commenter said about documentation nailed it. THat’s where I was going anyway. You have to figure out the patterns you’re _already using_ to deliver software to your clients. “Reify” them. Make them into “official” tools in your official professional toolkit :)

As you get more experience recognizing the patterns you’ve already discovered with reason and instinct. You’ll start to recognize patterns between tools, and patterns between problems. And you will see more opportunities for tools/techniques that solve wide swaths of problems.

I have a kitchen drawer full of really handy React hook I’ve written over the past couple years, for example, that are generalized enough to work anywhere.

It's funny you say that, as I'm starting to draft up a common core library (or SaaS) to share common techniques between projects and to streamline some of the constants.

FYI, I'm a data scientist that does front-end development as the reporting tool. Most of my projects involves creating some data analysis pipeline and then putting a frontend on it.

Can you refine the categories of projects you work on, and associate certain workflows with them? “Oh this is a ‘heavy PII data’ pipeline, I need my PII tool kit for this.” (Edit: I mean, for yourself, not here in this thread right now)

I’m basing this on how, when I want to learn about a new thing, I have the most success when I start by googling, “What kinds of _____ are there?” Just seeing the categories of the thing I’m trying to learn there are helps me a lot.

So maybe developing your own descriptive language for the kinds of projects you generally undertake would be a … well, now I’m just circling back to the documentation thing.

You seem like you’ve got your stuff together!

You want agile and test driven development.

Once you have a working prototype you collect user stories to modify the prototype. Clean up your code and move it towards solid principals using TDD.

TDD, Agile and Continuous Development all of them are failed ideas and best kept to academic musings.

TDD - write test before writing code. Only problem is coding is as much as discovery as implementation. You almost never construct the right structure, functionality and system the first time. It's only after you have played around that you start to understand the system and close in on something concrete. Writing tests in beginning would require you to visualize all the steps, instead of try and see approach. And you can visualize only so much. So TDD will prevent you from approaching to good solution and cause lot of wasted work. Tests should be testing integration or if unit tests then after code has been finalized.

Agile - this literally serves no purpose. If you are done with task you can move onto next task. Scoping some set of tasks to some duration of time is pointless exercise as you have no idea how much time tasks are gonna take. Estimates are notoriously bad. Quality suffers, tech debt multiplies and developers become impassionate who have no agency over their work. Agile creates these crude deadlines which means you add this artificial cool down and warm up time. If you finish early you can't start looking into something next, as you don't who will get the ticket. If you are bit late now next sprint is even more loaded.

Continuous Development - going back to 1st point, code is experimenting if you are pushing all your changes live 24/7 into master branch then you're going to end up in situation where the old code which you have decided to refactor/discard is now being used by someone else and you need to touch code that you had nothing to do with and have no context about either. If you had pushed only finalized changes then if others were to use your code they'd know it's stable. As for merge conflicts, only difference between CD and branching method is that in CD you just trade solving merge conflict at the end to solving it every day. Git history becomes so convoluted that you don't even know how to undo it because 3 peoples have been working in same file and pushing every few hours.

Sounds like your tests are to detail orientated.

Tests should test a specific business goal. You know what the goal is before you write any tests right? Otherwise why are you writing any code?

I don't know what world you live in, but business goals are never "test if a function returns an array of users with their comments sorted by likes". Business goals are more like user should be able to view comments on their profile. Which several layers of abstraction away from what coding is.

With regard to TDD I have been recommending many videos from this channel lately, and this one seems apt https://youtu.be/yfP_v6qCdcs

OP I agree with malthuswaswrong. About TDD, not malthus, I do not know who that is.

Thanks. I do aim for high test coverage, but still not driven by the tests.

Actually... There are two kinds of tests: tests that encode institutional knowledge and tests to make sure things work as intended.

In your case, you need only the later.

As others have said, I think it is not good to prematurely optimize a solution with too many unknowns.

I think a better approach would be to ask the client about what parts of the system are likely to change. If they are not good at thinking about this in a general sense, you should ask that in a more specific sense.

If you can get a better sense of what will change, it guide a little bit more of the design.

you just described 99.99% of software projects. There is no magic bullets. Keep the iteration going and don't overengineer.

Every software evolves but what I took away from op's post is that it starts out so small with just 2-3 features that you don't have a "story" after which to build your product and so there is zero "what if".

Example: "You can edit text" doesn't say if it will be Word or Visual Studio or Google Docs. You'd probably build different things for that and would already have a backlog of "nope, not yet" ideas where 5% will actually be asked for and will be implemented later.