That won't scale: Present cost vs. future value

Certain things, like using a database instead of a flat file when search is needed, or going with a linear algorithm instead of quadratic, are hard to do wrong.

Deep assumptions about the problem space are hard(er) to do right. This is why the architecture should remain somehow flexible.

Experience in getting it wrong a whole bunch in the past, though you'll probably get something wrong this time too.

It helps to know how far you can get with various approaches. Knowing that sqlite actually manages just fine for things up to X or where the reasonable edges are for using (say) a db for tasks rather than rabbitmq. Then some experience swapping things over. Been on a project upgrading from sqlite to postgres? Postgres on a single node to something larger? You'll have a better idea of how easy or hard these things are. Sometimes it's small things, like going with sqlite but being aware of the differences so that later the migration path is easier.

You can also sketch out architectures, and see what does or doesn't change if you swap out a component/step. I'm probably going to have something do a basic iteration for comparing vectors right now because

1. I know the scale I'm at means that it's going to be near instantaneous

2. For a larger problem I know I can keep the same interface (find closest X) and swap in various other tools I know exist

3. Adding those tools should be possible but due to some constraints probably at least some hassle

So I have a simple quick solution, but know how to change it later. I also have identified why I'm not going for the "proper" solution straight away, what the benefit is.

However, I'm going to keep the interface as "find closest to string X" not "find closest to vector X" because I've seen the issues that arise when multiple different things care about exactly how the vector is constructed. I've been through that kind of thing which dramatically slowed deployment of updated systems. Perhaps in those other cases it wasn't particularly avoidable but the problem feels like it has the same shape (hard to describe but problems and solutions feel like they have shapes to me, I hope that makes sense regardless of how you picture them).

These are some pretty small examples I know, but hopefully useful as approaches that can be communicated without a large background in a specific project.

My main point is where I started - experience making these mistakes. I am quite aware of deadlocks because there's nothing that teaches the lesson about lock ordering quite as well as seeing a robot freeze mid-task in a demo. When mentoring, people seem surprised when I solve some of their problems immediately, it's not because I have quickly solved it from first principles but because I remember making the exact same mistake 10 times.

> How do you know you're getting it right early on?

I think that's definitely the key question.

If it's the sort of situation where you've been given all the requirements up front - which does happen in certain domains - you can take the full-blown requirements engineering approach, and validation of the product will be built into the development lifecycle; it should be be effectively impossible to deliver an invalid product. (You might well go broke or lose all your team on death-marches trying to get there, but if you do, you know it'll be formally validated and signed off by the customer).

But of course, this is expensive, doesn't preclude delivery of a brittle product that can cope with change, and, moreover, just won't fly for situations where the requirements aren't fixed at the outset and the business is relying on dynamic interaction with the development team to evolve the product. So, for those situations, we have, what? Heritage, heuristics, trends. And the results are, to me...unsatisfyingly contingent. Why is the software shaped this way, and not some way?

For my $0.02, my projects are typically in between - there are some requirements provided, maybe they are a bit vague, but we largely know what we are trying to build, and we do some design to figure out what the big pieces are and how they ought to fit together. I've found some real success in approaches that subject the design of the software up front to the stresses of potential change, like those given by Juval Lowy (IDesign, Righting Software) and Barry O'Reilly (Black Tulip, Residuality Theory). And to subject a design to stress, of course first you have to do some system design! If you're not at the stage of being able to put together a design, I think it's likely more a prototyping or an investigative programming kind of activity, which to me has a different goal.

> I don't think it's cheap to do early. The costs are in people writing, drawing, talking, and often debating, rather than doing iterations on a basic working solution, and the outcome usually doesn't really work and needs to be redone in a year or two when more information is available.

It is, for vast majority of apps.

All you need to do to make sure your app running on one server can be run on 10 i.e. using actual SQL transactions to manage data, not in-program-memory locks. Then you can just get fatter DB server and boom, you're 10x

Above that, sure, the "bigger server" starts to be expensive so you might need to start thinking about sharding, or moving some data off to separate instance (separate search cluster for example), or adding cache here and there, or, eventually, rebuilding the core of it for performance but that's 100x territory

"Bend" is not a good description of the intention here. Good code should be easy to test effectively and interfaces are a common way to achieve that. We should absolutely design our code with testing in mind.

For example, you have a repository for some entity as a dependency of a service and you would like to test some behavior of the service when the repository fails. In this case, the service can depend on an interface of the repository instead and the test can create a stub that implements that interface, whereby you can simulate the error. If the service depended on the actual repository, this becomes very difficult to do.

Your tests are part of your code. Your implementation code should not make writing your test code difficult.

It's kind of unavoidable once you're trying to do unit tests with mocks. You either end up with interfaces for everything and a dependency injection framework, or you end up with some very non-isolated tests that take ages to run because they need inseparable parts of the whole real system.

If you start with AService in module A, other modules need to depend on A to use it. You get into circular dependencies trouble when A needs to use a service from another module B that itself or through a dependency needs the AService. One solution is to split it into an AService interface you put in a new module, call it e.g. A-api, and leave the implementation in A which now depends on A-api. Do the same for B/B-api. It's a lot less likely that your -api modules will circularly depend on each other because they only contain interface definitions, so A can safely depend on both A-api and B-api and B can safely depend on both B-api and A-api.

A more concrete example of an A and B that need to interact with each other might be a user account service and a (financial) transactions service. (Why not let them live in the same module? That can also be a solution, but as the scope of each grows and especially if separate teams end up owning each it can make sense to split them and enforce some boundaries via the interfaces.) Another example could be a notifications service and several of its clients, like whatever is handling user posts kicks off a notification event and asynchronously the notification service processes it and may need to reach around to call back into other services that could be part of the sender's module. (Passing simple lambdas may be an adequate alternate solution too.)

I think that's part of the issue. Once you're using a text file, adding "one more feature" to the text file implementation is always easier than switching to a database.

So you add your tab-separated value. Then you find out that in some cases the extra field you're adding can contain a tab. So now you need to be able to escape actual tabs in your tab-separated file - because once you have the tab-separated file, adding an escape is still easier than switching to a database.

Cut to: 3 years later. You now have optional fields, a key-value implementation, locking, and two or three bits of code that work around the edges of the "proper" API to access the file directly, because at every stage that was easier than creating a "real" database backend. But now its harder to change than it would have been to do the right thing at the start, and the sunk cost fallacy means that everyone's brain is resistant to throwing away all the work that's been invested in the current solution even if they know it's the right thing to do, and even though whenever you onboard a new recruit they always fail a SAN check and take a couple of weeks to recover from the horror of your codebase before they can face it for reals.

And implementing the next new feature in the text file still looks easier than replacing the whole implementation with a real database right now, so let's put that off for just one more release cycle, OK?