Applying “make invalid states unrepresentable”
kevinmahoney.co.uk
kevinmahoney.co.uk
From a practical point of view it might be advantageous if you maintain only a single contract with a customer at all times, but that is a business requirement which might be changed in the future.
I mention this mostly from experience: Multiple times I've designed systems where I've reduced the representable states to the minimum, and when some requirements change I realize I have to re-design the full system.
The new presented representation might make sense in this situation, but I'd be very wary of taking current business practices and make all other alternatives impossible to represent. It's a balancing act of course as you can go in the opposite direction and make it way too flexible.
> This poor choice was not just a theoretical problem - gaps in contracts were found on more than one occasion, requiring hours of engineering effort to hunt down and fix.
I'd like to hear more about what happened here. Was the problem that the default contract was not re-applied correctly? If so, changing the representation might not actually solve any problems — it make actually make it _worse_. A renewal of a contract typically involves some automated process where other services are involved (payment, invoicing, emails). The previous representation (with explicit start/end dates) made it possible for you to verify that everything was correct and lined up.
Yes, if requirements change, you change the design and code to support the new requirements.
Compromising the consistency and maintainability of the current design to accommodate a hypothetical future requirement change is a bad trade-off IMHO, since you can't predict the future. A requirement change may happen in a completely different direction than the one you anticipated, and then you have the worst of both worlds.
It is better to make code maintainable than making it flexible.
That's one of those statements that makes sense, but is often not true. Very rarely does a client comes to me with a feature request that does require a pretty significant design change, but most of the time they're changes that were foreseen.
Using this current article as an example, I love the way that they're storing the intervals to guarantee that they can't overlap. That's awesome! What I would likely end up doing, though, is use that as the underlying representation but still return individual interval objects through the query API with a start and end date on each interval. That way, if the "only one at a time" rule changes, the changes required are localized.
The article addresses that concept:
> It is sometimes still useful to represent the periods as a sequence of start and end dates. It is trivial to project the set of dates in to this form. As long as the canonical representation is the set, the constraints will still hold.
How the model you present to the user is represented in the database is an implementation detail.
"Never think ahead" is obviously not good advice. There's no silver bullet here--we have to think about how likely future scenarios are, and plan for them based on the business context and needs. Many of them are unlikely or too costly to do anything about...and many of them aren't.
But trying to anticipate everything in the first iteration is not the solution. The solution is to write maintainable code and apply each change with the same discipline and thought which was used when in the initial iteration. Follow the boy scout rule: After any change, the code should be in a better state than before.
Now you have to work around your simplified design because you decided that you didn't need anything more.
I suppose if you were cynical, you could claim that if it's painless no one sees how important you are. And then you wind up leaving the company, because they think everything is easy and don't provide you with the autonomy to achieve what you need to make their system work. And then they discover that it's actually hard.
Code and representation (i.e. schema) are vastly different. In my experience it's takes an order of magnitude longer to change a representation than to change code. Once there are multiple services/tools which works with a representation you typically have to support both the new and the old representation at the same time (since you can't rollout everything simultaneously).
> Compromising the consistency and maintainability of the current design to accommodate a hypothetical future requirement change is a bad trade-off IMHO
Designing a representation which can handle possible requirement changes does not necessarily mean "compromising the consistency and maintainability". We have great tools for ensuring consistency (e.g. transactions and constraints in SQL databases), and I don't exactly see how this new representation is more "maintainable" than the old one (although we don't have all the information in this article).
On the other hand, some may say YAGNI.
DRY is too often treated as purely (or principally) syntactic, when that's actually much less useful.
The business also tells you that there are never two contracts running at the same time. But are you actually going to believe that? Is this condition really "impossible?"
A vital and necessary factor here is whether the system being designed has complete control of the creation of contracts. This is perhaps taken for granted by the author, but it's too important to leave implicit. You have three choices in a situation like this: make it impossible for contracts to overlap, model it, or don't model it and accept the consequences. Depending on the frequency and the consequences of the assumption being incorrect, maybe it's acceptable not to model it. Maybe not. My point is that you can't assume something is impossible unless you can actually prevent it from happening, and the author should not have sidestepped this part of the analysis (though possibly they meant for it to be understood that contract creation happens through this data model.)
> Also don't forget that this is only an example.
The problem is that this is an example meant to illustrate and justify a rule of thumb, but it's extremely, extremely simple. How often do you deal with requirements that are this simple, this mathematical? Is this really the kind of example you want to build a rule of thumb from?
Realistically, when I hear requirements like this, I assume they're wrong (very common at the beginning of a project) and I get together with the product manager and ideally a domain expert representing the customer (if the product manager isn't too territorial about that being their job) and figure out what the hell the actual requirements are. What if a customer has a contract to rent 10 units of space at $5 and in the middle of that contract needs more 5 more units but the price has gone up $10? Do you tell them they have to cancel the existing contract at $5 and pay $10 for all their units if they want to add some? Or give them the new units at the old price? Or is it okay to represent the same customer by distinct customer records?
I do like the principle of making invalid states unrepresentable, but I would like to add two supplementary principles:
1. Oftentimes what the business tells you about the data they produce is purely aspirational.
"There will never be overlapping contracts," often means, "We swear we're going to stop creating overlapping contracts, and this time we really mean it." You have to follow up with questions like, "How often have we had overlapping contracts in the past? When was the most recent occurrence?" You should even ask, "When do we anticipate signing the next one?" A logically-oriented software developer might expect someone to take offense if you respond to "we don't sign contracts like that" with "Do you have any currently in the pipeline?" but this is a totally normal kind of question to ask.
2. When users give you a rule in their business requirements, they often take it for granted that the software will handle exceptions to the rule gracefully.
They don't necessarily appreciate how bad things can go in software when something "impossible" happens. When they say, "Contracts will never overlap," you have to say, "What should happen when they do?" If you are talking to a mathematician or a programmer this might come off as questioning their competence, but most people will not find it unusual at all or at least will appreciate that the question is motivated by experience rather than disrespect. It's not like a math problem in school; it is legitimate to question the givens.
If things go well, business data outlives business applications. After years or decades, it can be a major pain to figure out all the "secret" values that aren't actually in the data.
Agreed and yet for many systems, that flexability still eludes.
One example I personally experienced was changing a phone contract. The contract had run for many years, so could be cancelled any time with one months notice. The new plan and contract was much better and yet a limitation played out doing this. Ended up that the systems at the telco was unable to activate the new contract until the old contract had ended. Whilst a new contract could be physically signed in a shop with a start date of the day of signing, and logged into the system. The provisioning backed was unable to activate it until the old contract had ceased as you can't have two contracts for the same phone number.
That I do believe is a case of - whilst some things can run in parallel, others are locked to a single dependant resource.
But every rule has an exception, it is with good design that you limit those exceptions impact.
It appears to actually be some sort of contract_period or contract_duration, and probably has a link to a real "contract" object somewhere that contains the real meat of the concept. But it's hard to tell what's literal and what's the author trying to simplify the example for us.
I've found the fundamental principle that helps to keep the system extensible is to: make your system model the real world accurately.
This involves building the system concepts closely mapped onto real world concepts without taking shortcuts.
That way when the requirements change, all the fundamental pieces of the system stay valid, and only the piece that is changing tends to need updating.
This helps to void the problem you mentioned of needing to re-design the full system, keeping the system extensible.
Also in generally the principle of "make invalid states unrepresentable” ends with the realization that only Idris can properly do it, which is not pragmatically useful.
I have no idea about the commenter here, but in my direct personal experience, the "those who don't" developers tend to regard software design and requirements to be toward the "write-only, then try to never change" end of the spectrum.
I get a lot of "are you crazy? We can't change that!" type of pushback from the those-who-don'ts, regarding functional changes that should be reasonably manageable to my testing/typing mind.
Basically this is what event sourcing tries to solve: it lets you change your state representation to reflect new requirements, because you can always rebuild everything from the event log.
That reminds me of an interesting point I saw in a book whose title and author escape me. He said one of the reason you encounter so many bad designs in Java programs is that many new Java programmers look at the design of Swing to learn good design.
He wasn't saying that Swing is badly designed--but Swing is a framework/library, not an application. What it takes to be a good framework/library is different than what it takes to be a good application.
If you are writing an inventory management application you can design your tables and data structures and interfaces around things inventory management applications need. If you are writing a medical billing application you can design around what medical billing needs.
If you are writing a framework or library that might be used by inventory management applications and medical billing applications and all the other nearly infinite kinds of applications people will write you need to keep it very general and flexible...but you also have to keep it fast and not too bloated. It's a much harder design problem, with different best practices for what is good design and what is not.
1) Don't advertise what you're not selling
2) Don't sell everything that you've made
It's possible to write software that has a public contract that allows only a subset of states that the internal system allows. You can use this to support one customer that has a requirement that is mutually incompatible with another customer's, but you can also use it for migrations. One should be able to create an API where the legal values in the system are strictly limited, but the data structures and storage format may have affordances to support migration.
That doesn't necessarily solve the problem of communication between services and migrating these changes into a running system, but it's a useful tool. If you've ever had a coworker who insists on your service call diagrams looks like a tree or an acyclic graph, this problem is certainly one of many reasons they may be insisting on this. With a DAG there is an order in which I can deploy things that has a prayer (but not a guarantee) of letting all of the systems understand each other during each increment of deployment.
People have come up with alternative solutions for this problem by employing sophisticated sets of feature toggles, and in some ways this is superior, but it trades the number of steps (each of which has a potential for human error, and consumes calendar time) for increased reliability on average.
The timeline example is actually very telling. A lot of times you’d actually want to encode overlapping time periods at the edges.
You’d be laughed out off a meeting where a business asks about this and you smugly explain how it is unrepresentable.
I guess what I’m saying is that it might be worth over designing your system a bit to leave you some wiggle room, unless you have hard guarantees that something should be “impossible”
If any of the requirements change, it's easy to either add more predicates or relax/even outright remove them.
I don't deny there's a certain art to that, and I can't explain it all myself. But I don't even necessarily mean amazing clever type tricks like you might see in Haskell or something, I mean that something as simple as "I've changed the definition of what a 'Customer' is in some fundamental manner, so I'm going to rename that class to 'CustomerNew', use the compiler to point me at every single place that breaks, audit it, change the local name to CustomerNew, and then, once everything is fixed, use my IDE's rename feature to rename CustomerNew back to Customer before my final commit". Many times you can get by just by renaming a field or something to similar effect, but in the worst case you may need to audit everything.
It's one of the more tedious bits of the job sometimes, but net-net this can still be a timesaver, if you account for the full cost of the trickle of bugs this sort of thing can prevent.
On the other hand, if you make your internal representation support more complexity than what you hand out to your users, there arises the question of how you simplify (probably lossily!) or break when you wind up with data that can't be expressed with the simpler interface. But you might well be saving yourself a costly data migration down the line. Also a decision with trade-offs that could potentially go either way.
Customer: 1 - 1: Set(Dates)
Customer 1 - *: Contract: 1 - 1: Set(Dates)
SELECT event FROM events WHERE endDate<"2021"
Whatever additional complexity you
add to your codebase to query end
dates.https://dbfiddle.uk/?rdbms=postgres_11&fiddle=50e6a963cd1db0...
(YMMV, obvs.)
SELECT event FROM events WHERE startDate < "2021"
It gets uglier if you need to find the contract valid on a certain date based off of a join.
These issues can be covered up with code as it will a easier to have reusable functions, but it makes the job of a data analysts much more difficult and error prone.
I think a large contributor to the problem is story-oriented development, where all that matters in the sprint is "getting it done" and not looking at the broader context.
To make unrepresentable states practical, Scott Wlaschin has an excellent write-up here (0). His book (plugged in that article) is also excellent.
[0] https://fsharpforfunandprofit.com/posts/designing-with-types...
I think you have a point here. This design offers much better safety, comparable to "parsing instead of validating". But it requires up-front design. And that is indeed "verboten" in modern software development management style.
Why is it "verboten"? I think it has to do with two fundamental concepts of scrum et. al.:
1. Stories that are "ready" just need to be "implemented". The implication is that the developer does not design and everything is orthogonal. There are no interdependencies, maintenance effort or non-functional requirements.
2. A story that is "ready" focuses solely on the desired outcome for some selected examples. There is no generalization of the examples and consequently little to no abstraction.
I think these issues stem from the fact that Scrum et. al. are intrinsically tools for managers to isolate them from the complexities of software engineering. Every metric of scrum, for instance, like "progress" or "definition of ready" is essentially empty of meaning for software engineering.
If a system needs to be designed in a particular way, do so. That's how long it takes and that's why in planning you discuss how it will be designed.
The design of how you're going to build the system is taken care of before the task is split into easily digestible bits that meet a definition of ready.
If you need to build it before you know how to build it, scrum allows for research spikes into this. A focused, timeboxed interval that so you have the ability to estimate the actual difficulty of work to be completed.
Details will always matter, but management's paycheck depends on not understanding that.
I think that's exactly backwards. This kind of overcomplicated representation usually happens because people put too much effort into designing their representations up front. If you follow story-oriented development and only implement the parts they actually need to get the current task done, you never end up with these wasteful extra states because you never actually needed them. But people think that planning before coding is somehow virtuous, and then they're tied to following those plans.
- A team spends too much time over-engineering a representation for something that could be more easily maintained using a simple model
- A team spends too little time considering the edge cases with a representation because they feel pressure to deliver the feature within a short space of time
I find that both types of representational debt are common.
Also have experience fixing story-oriented development with dirty workarounds, because major refactorings were also required, but not desired by whom was paying for the stories.
Both ways I didn't care, it was money on the bank anyway.
personally I think that's a better trade-off than implementing something complicated up-front that you don't know will work and ending up with flexibility in the wrong places, leading to tech debt and hacks.
{ok, Ok} | {err, Err}
There's a PEP but I stopped following it because the discussion was a mess. And it's apparently now split into 3 different PEP, I don't know whether that's an improvement or not though.
Furthermore Python's dislike of HOFs means you can't really do "monadic" processing as you'd do in, say, smalltalk where your "variants" would really be subtypes with cool higher-order messages. So you're mostly just adding indirections.
# data Pair a b = Pair a b
def pair(x, y):
return lambda f: f(x, y)
# data Either a b = Left a | Right b
def left(x):
return lambda l, r: l(x)
def right(y):
return lambda l, r: r(y)
v0 = pair(2, 3)
v1 = left(7)
v2 = right(9)
# case v0 of { Pair x y -> x + y }
print(v0(lambda x, y: x + y))
# case v1 of { Left x -> x + 1; Right y -> y * 2 }
print(v1(lambda x: x + 1, lambda y: y * 2))
I'll need to read more about it, thanks for the pointer!
It's serviceable, and even does static checking of total coverage of cases. I will say that it makes linters freak out.
Make your schema permissible and your code paranoid, it will pay off later. Build a data linter if necessary, but don't tie the schema.
I phrase this as: If it doesn't make sense to do math on it, it's not a number. What does adding one to a customer account number mean? Absolutely nothing -- you get a completely different account number. So it's not a number, but a numeric string.
Namely, when constructing a protobuf, theoretically, there might be two different ways: (A) first gather all the fields, and then construct the protobuf from these fields; (B) first construct an empty protobuf, and fill in the fields as necessary. The actual protobuf uses (B) - which is convenient in most cases, because when you start constructing a protobuf usually you don't have all the data ready yet.
However, with required fields, this means when you construct the protobuf it starts with all required fields missing - i.e., an invalid state!
I'm not sure what's the best way to fix it, because it would be infeasible to rewrite all the code to gather all the fields and then construct the protobuf - also it will be hugely inefficient in many cases. However, I feel the "no required fields" rule is essentially a null pointer (the "billion dollar mistake") in disguise - the actual problem is that the API doesn't enforce type safety.
Imagine you have an innocent `required` field. You have a producer and a consumer of that field that communicate over the wire. (or instead of the wire, imagine a database).
You send or store an instance of that protobuf. Now let's say that you want to make the field optional (or remove it). With an already-optional field, this is easy. You stop setting it, and maybe eventually you clean it up.
With a required field, however, you can't do that. If any of your clients don't have the newest schema version, you can't unset the field (so imagine that you support mobile clients who may never update). Or if there's middleware you don't know about that introspects your proto. Even if you do the dance right and update your server and client before not setting the new field, you could crash outdated middleware that you didn't know about. Whoops!
Or with the database, you now need to dual write or something complex because if you need to roll-back to an older version, you'd be unable to read the protos that don't include the required field.
Required doesn't do well over time. It has nothing to do with setting the values.
Changing required to optional isn't a magic fix for protocol compatibility. If it were (for your limited use case) you can just make that change to the protobuf client side as it doesn't affect the wire representation/interpretation.
Right, there's the rub. `required` means that anyone who deserializes your proto falls into this category. That's a much larger group than "anyone who reads a specific field". So the list of clients now is forced to include any and all middleware that may read your proto (imagine a routing layer or some kind of analytics system or whatnot).
(Note also that there's lots of ways to make reading a field that is empty fallback to doing some reasonable non-catastrophic behavior, required doesn't let you do those things).
Making invalid states unrepresentable is basically the process of taking human-checked invariants and turning them into type-checked invariants. This reduces the likelihood of bugs and guides humans to use the system correctly.
Required now means requires forever because people can't migrate safely. But technically you can change a protocol descriptor from required to optional, which is invalid (usually, in a distributed non-transactional system (the common kin) but nothing stops you from doing it. So why not make required forever? Well, do you really want to commit to anything forever?
For example, in the case of the message bus they say "And even though the message bus doesn’t care about message content", and later on "The right answer is for applications to do validation as-needed in application-level code." Strict schema and validation is most helpful for application developers, not some middleware routing code. Was it not possible for them to write a parser that doesn't fully validate the message for use-cases like this?
Required usually cannot be deprecated.
Because to be sure of that you can remove it, you need to be sure that every storage system and every piece of middleware and every since thing that links your proto anywhere in the world that you might care about is upgraded, otherwise if they encounter a new message they'll crash.
If you only have a single client and server, and you control both, this is doable. If you don't have that though, you cannot.
In that case, applying the remedy of the first example (a set of dates, and inferring that every 2nd (even zero length, to account for adjacent fixed) interval will be default,) introduces another bug where if you lop off any random date in that set or list, you invert everything.
I love the concept represented here, it is akin to normalization as in...simplify the representation so no redundancy is introduced as this usually leads to better results...but it seems it's no guarantee of better results.
But maybe that's just because the "model" we are simplifying from was not an optimal representation. Perhaps there's a better model of the second example that doesn't end up with the defect of the first example.
I really like this article but am struck by how something that I wanted to be almost a silver bullet trick for modeling, ends up being a mass of compromises mired in tradeoffs that doesn't show any clear way forward in the general case. Still, probably a good rule of thumb, but I guess this rule is not optimal...as it can have so many unworkable misinterpretations/misapplications.
It would be cool to see a list of, like, "Programming Heuristics", ranked by decreasing general applicability, of which this rule was a member somewhere far down the list.
This is a good point, and something I usually describe recoverable problems versus non-recoverable problems. If I make start/end dates in the first example instead of just a set of start dates, then I can always create application-level or database-level constraints that don't allow either overlapping or incomplete segments. When business rules change, I can delete the constraints and update the business logic as necessary with no change to underlying data structures.
However, if I miss implementing a constraint and it erroneously allows overlapping or incomplete segments, I can easily run a query to identify all such invalid entries. Then I can then investigate and decide how to fix them.
However, if I go with the start-date-only set-based approach, and miss implementing a constraint, and it leads to a deleted date creating incomplete segments... I'm screwed. There will be no query you can do to identify incomplete segments to investigate or fix, because all segments are assumed to extend to the next start date. You can irreversibly lengthen one segment by deleting another, due to a forgotten constraint preventing you from making the change.
These could both be errors on the developer's part, depending on the requirements at the time, but one data design may lead to more non-recoverable issues than the other. Add in the flexibility of the former approach, and I'd probably be more likely to implement the former approach than the one proposed.
+1 for this! Anyone got good links to share?
Dates are a rich vein of examples. Some users will happily consider “25th December” to be a date, without any year, because it might be the name of a folder in which they store their Christmas stuff. More seriously, genealogists or historians may want to record “25th December” (again without a year) as the date of a photograph because they can clearly see it was taken on a Christmas, they just don’t know which one. The naive developer would just slap a DateTime type onto the system and feel good about having avoided malformed input.
- estimated dates
- calculated dates (e.g. someone was 30 in 1870, so he was born in "calculated 1840")
- unreadable or unavailable months or days (typically recorded as 1980-00-13)
- time ranges with all of the above as boundaries e.g. "after 1760-03-00 and before calculated 1800"
- plainly incorrect dates, but that's what the document says (1865-02-30)
- no dates (some software tries to enforce putting in some data in for whatever reason)
- dates with unknown calendar
For that matter, someone might want to do a text-based search on dates: "This damaged photo shows 196_-_1, which could be at least 20 different months..."
In my area of research death records are typically a bad predictor for date and location of birth. When the records were managed by the churches (until after WWII pretty much), priests did not want to pester grieving family for the exact date of birth, so they relied on approximate age. Naturally the subject of the death record would not typically point out errors. On the other hand marriage records required looking at the actual birth records (sometimes mailed across the country), as these contained notes on all marriages of the individual (this was done to ensure monogamy).
Is it on the genealogist to consider this confidence when specifying the date of the event, or should the software intervene? Due to complexity, the former is the industry standard. But maybe there are some brave (or stupid) people who will try to take it into account in the future.
Like you say, a "folder". But the photo file's metadata will either have a complete DateTime, or none at all, unless there is some sort of camera that is able to know what day of the year it is without knowing the year! Which due to things like leap years is impossible.
If you and your client disagree about what constitutes a date, that doesn't mean "your client is wrong" it means "you need better communication. You can't fix this problem by requiring everyone use consistent definitions. Instead the solution is to check assumptions as often as possible.
Just a small note, as we're talking about assumptions, but this is clearly untrue. Consider - for decades, humans wore watches that knew the day of the month, but not the month - you were just expected to turn the day forward on the 1st day of months following non-31 day months. Similarly, we can imagine disposable cameras that ask the user for the current day on first use and simply assume leap years don't exist, and require the user to correct the date for any leap years. You might call this a silly design, but systems often have to interact with external systems that have silly designs. I believe I actually owned a toy PDA (a device for a child, not an adult) that did not have a year back in the 2003 or so.
a Time Period object of ( Date, Duration ) would fix the first issue, and a TimeLine of ( Date, Duration, Duration, ... ) would fix the second one (assuming unsigned Durations!)
One complication it's potentially missing is exactly who's days we are talking about i.e. is it days starting in GMT or UTC or EST or whatever the 'suppliers' or the 'customers' timezone is, are we actually talking about some day concept perhaps from start of business. Representing this as a datetime start / end certainly makes it possible to represent these concepts, if perhaps not making anything else particularly easier.
When implementing this in practice, you probably want to use a set implementation that gives fast ordering results. But that is a performance consideration. Not a data-representation consideration.
The timeline is {date1, date2, date3, date4}. Let's say you have 2 periods, date1 - date3, and date2 - date4. Period 1 can be represented as {date1, date2, date3}. Period 2 can be {date2, date3, date4}.
Am I understanding this correctly?
"If the customer doesn’t have a fixed contract, it is assumed they are on a default contract"
No. Don't assume, specify. Explicitly.
If this is part of your specification, have a DefaultContract entity of some kind somewhere. And don't call this table just "Contracts", make it clear that those exist in addition to or overlay a default contract.
It might sound like overkill, but in my experience in business application development, one of the single largest and most painful sources of errors and refactoring headaches are implicit assumptions in the data model.
This is the OO mindset described at the bottom - the odd compulsion to have a reified entity for every concept.
A lot of people have missed that the representation you persist doesn't have to match the representation you present. In this case, don't store default contracts, but present them e.g. via a database view.
And yes, if there's something as fundamental a concept in the business model as a default contract that's in effect when no other contracts overrule it, then IMO it damn well should be represented explicitly in the persisted data model.
I didn't talk about the specific nature of the representation. The important part is that the intent of the data should be explicit - data lives longer than code.
I've seen far too many DB schemas leaning too hard on implicit assumptions and inferring information that lead to hard to understand data models, unnecessary complex (and hard to optimize) data access (no matter the paradigm), and well, lots of errors.
Yes, but not in the database table for contracts. That's how I read this part of the post. I would expect the assumption that if the customer doesn't have a fixed contract, they are on a default contract to be encoded in business logic somewhere in an application that uses this database.
That's exactly the kind of harmful assumption I'm talking about. Harmful in that when people act on that assumption and actually implement it that way.
How a default contract might be represented may vary, and of course it is in no way required or even sensible to be stored as a row in the contracts table.
But to think that it is such a fundamentally different kind of data that it should be represented apart from the rest, in a different system, in a different layer, in a completely different form of representation - this is where madness lies.
But you're saying it does need to be represented somewhere in the database? That putting it anywhere else is harmful?
Can you elaborate? Why is it harmful?
The concept of a "contract" is a business logic concept to begin with. That concept is represented in the application already. How (or whether) the data associated with a particular contract is stored in the database is an implementation detail.
Or maybe you start reporting and want reports on contracts vs default contracts.
And then things get messy. Because the real world gets messy.
The end date of a contract is a property of that contract, not any other.
Forgetting all other contracts for a moment, what do you need to know about one contract? There should be a straight forward way to query that contract on its own, with a query that represents a tree through tables in the database. It should not become a graph, with the potential for cycles that graphs allow.
And I get the business requirements could need no overlaps, but gaps are clearly possible if a customer leaves for a while and then comes back later. Does that person need to then become a new "customer", because you don't allow gaps? And then are your customers' PII only allowed to be registered to a single account? Comcast has been a huge pain in the ass in years past because of moving, gaps, and email address reuse.
What's needed here is to sit down and think REALLY hard about how the system works and what some of the terms are that are used to describe things. And also to have an intution for when someone is using imprecise language. GP cited use of a "Schedule" object as a way to enforce the invariant. That might represent a block of intervals by a sequence of times.
The less precise you are about what's needed and how the system should work, the more of a soupy mess you're going to build.
People gravitate toward OO because OO makes it feel like you've got a lot of conceptual clarity. You can get back to that familiar "subject/object" dual we love in English. But the problem is actually that, although people can pick subjects and objects readily from a sentence, they might not have much luck picking the important subjects and objects from a sentence. And that's the problem we really need to solve.
You take requirements and make system exactly right to fit these requirements perfectly and don't bother with any other concerns.
When you design this close to the requirements you get better, faster and more elegant code that's easier to understand - but when requirements change you have to do much more work to adapt. Suddenly a state which was previously invalid is valid, or a part of system that only needed one kind of input needs 4 different inputs from separate parts of your code. Have fun basically rewriting your program.
That's IMHO the main motivation between differences in functional and OO programming - how close to the requirements you want to design your code.
C (very loosely typed language) code is usually pretty close to the requirements and very strongly typed functional languages (like Haskell) are often "make DSL and write the specification in it, then run it".
Meanwhile object oriented languages often have pretty strong typesystems and cultures of using them extensively, but they also encourage designing with margin for changes (and thus using lots of layers of abstractions instead of just implementing the specification as elegantly as possible).
The obvious problem is that those three props can easily get out of sync if you make a mistake in updating them, and the equally obvious solution is to replace them with a single prop (e.g. `show` or `mode`) which contains a value from some enumeration — maybe just the name of the thing to show, or a numeric ID that is given meaning elsewhere, or (galaxy brain) a component. That way the invariant is maintained strongly and automatically by the representation itself.
This example sounds vacuous — who would ever use three booleans in the first place? — but in practice it’s very easy for UI code to incrementally get into this mess over time without anyone noticing. The situation is also often more subtle, e.g. the invariant is more complex than “exactly one flag is true”, which makes it harder to spot that you can model all the valid states with a finite enumeration.
For those of us developers who are mere CRUD peons I think it's the most important factor in system stability that is mostly negected; either in favour of speed of iteration (NoSQL) or checks at the application code layer.
As I'm increasingly coming to appreciate, systems without enforced integrity at the database level are a breeding ground for bugs. You can add checks in application code but all it takes is 1 bad commit, or 1 check that slipped your notice and now you have bad data and all future code in the system needs to support and work around the bad data. With foundations of sand even the most elegant structure in application code is doomed to a short and catastrophic future.
As other commenters mention hindsight is 20:20 and you won't always know what the constraints should have been until after the fact, or the constraints might be wrong. But the 'trendy' development practices treat good old fashioned SQL constraints and data integrity as decidedly unsexy, to the detriment of a lot of systems.
MySQL didn't even have check constraints (well, actually apply them) until version 8 which shows how ignored these things are. I appreciate the post is more about the fundamental design of the stored data but people are also forgetting unique constraints, foreign keys and all the other tried and tested tools which protect the most important part of most CRUD systems, the data, from devolving into an awful mess.
Keep the model sensible. Contracts belong to contracts. Add basic sanity to the model. The service that manages the data guards the data beyond basic data model sanity checks. Also, explicit is better than implicit.
Take the example of storing a continuous series of date ranges. If I only store the first date of each pair, I can never accidentally have an overlap or gap. But if my code has a logic error that incorrectly calculates a range, being able to represent it could throw an error. If that code error translates to an incorrect break-point instead, I haven't prevented a bug, I've hidden it.
The extra code and conceptual complexity spent to make type designs that disallow invalid cases is a liability, it comes with its own bugs, maintenance and huge risks of premature abstraction and brittleness in the face of changing requirements.
If it takes anything more than a simple enum-style menu of permitted options, then it’s a code small. Things like Scala case classes (especially with sealed behavior), or pattern matching against type constructors, or phantom types - these are all very bad ideas, where the costs far outweigh the benefits.
Most of the time you can just ignore enforcement of assumptions, and add a few assert statements plus lightweight unit tests and integration tests that generate an abundance of real world example cases - and achieve all the safety you need for a fraction of the code & conceptual complexity and tech debt incurred by false promises of enforcing correctness with type system designs.
I attacked the old 8-queens problem years ago as part of a contest in Byte Magazine. All the solutions published modelled the board as an 8X8 array with a 1 or zero to indicate the presence of a queen. They all ran slow and suffered from invalid game states confusing the algorithms.
My solution was to observe that only 1 queen could be in each column (all solutions require queens to not be able to capture one another, and they can capture vertically). So I represented the board by an array of 8 values, the height of the queen in the column represented by indexing the array.
Further, since only one queen can be in each row, the values were the numbers 1-8.
My solution then, was to seed the array with the value 1, 2, 3, 4, 5, 6, 7, 8. Then test if array[i]-array[j] == (i-j) or (j-i), which would mean a diagonal capture.
Simply permuting the values, searched a subset of board states that had to contain all possible solutions. And the permutation tree could be truncated as soon as for any (i,j) the test failed.
Anyway, the program was tiny and finished in negligible time. A pity I didn't enter the contest!
As with database normalization, it is a good idea to first do it as much as possible, and then denormalize again until it is fast enough.
Essentially, you keep the DB in a normalized state. You define views of the DB that you want. Then the DB keeps those views as tables for you, and the DB does all of the hard work of keeping those view tables consistent with the denormalized data. Essentially the DB does the atomicity, cache-invalidation, and cache-updating for you.
You get performance, and you get the certainty that invalid states are un-representable.
I guess the biggest blocker here is in automatically determining what fields do and do not need to be update automatically?
https://github.com/mit-pdos/noria
https://corecursive.com/030-rethinking-databases-with-jon-gj...
https://notamonadtutorial.com/interview-with-norias-creator-...
That said, it's an excellent, commonsense article that describes a highly usable approach to information architecture.
I also agree that OOP programmers have always considered their designs to "represent the 'Real World'™." In my experience, I use OOP constructs to represent many things that should never be exposed to the user (like messages, adapters, states, and state transitions).
There's the classic usability concept of the "Mental Model." That is the model that the user builds in their head, as they navigate the UX. These mental models can be drastically different from what happens internally, and a good UX designer can reinforce a desired model (which the user may then ignore).
If you think storing a list of date tuples is "OOP thinking", you have no clue what OOP really is. Educate yourself by listening to people who invented it, not Java consultants or FP zealots.
OOP is about interacting with things via interfaces and messages, rather than data. An OOP solution to inconsistencies of this sort would be an interface that either automatically corrects inconsistencies or throws errors when you try to introduce them. The whole point of OOP approach is that you're not locked into a single data representation, so, for example, you can improve how you store data without re-engineering everything in your system that relies on that data.
Whatever the customer signed is the authoritative contract, not some imaginary entry in a schema intentionally constructed to make it impossible to represent scenarios not envisioned by the developer.
https://fsharpforfunandprofit.com/series/designing-with-type...
For instance, one micro-application of it that makes a lot fo sense to me is the const-ness of variables in languages like C. Since a normal variable can be overwritten, and that affects the use and semantics of that variable, marking them as const whenever possible really helps in my opinion.
For instance, take this micro-snippet of code from Redis [1]:
int time_independent_strcmp(char *a, char *b) {
char bufa[CONFIG_AUTHPASS_MAX_LEN], bufb[CONFIG_AUTHPASS_MAX_LEN];
/* The above two strlen perform len(a) + len(b) operations where either
* a or b are fixed (our password) length, and the difference is only
* relative to the length of the user provided string, so no information
* leak is possible in the following two lines of code. */
unsigned int alen = strlen(a);
unsigned int blen = strlen(b);
unsigned int j;
int diff = 0;
So, I think the middle two lines should be:
const size_t alen = strlen(a);
const size_t blen = strlen(b);
Also please note that I have massive amounts of respect for Redis and Antirez, I'm not trying to say that the code is bad or anything, it was simply the first file in the first high-profile open source project that came to mind. Obviously this code works and has probably been more tested than most things I've written, again I'm NOT trying to somehow paint that program(mer) in a bad light.
Btw, changing the type (to me) to size_t is also an obvious, free, improvement since it frees the reader from having to worry about why the type was unsigned int to begin with. Also 'int' can be less wide than 'size_t', which again is probably not a problem in practice since the CONFIG_AUTHPASS_MAX_LEN is probably always going to be even less, but still. It's pointless complexity that triggers anxiety in people like me. :)
Does anyone have any questions?
edit: or related hangers-on, like entrepreneurs or "thought leaders"