Handling Null Values in Postgres
percona.com
percona.com
postgres=# SELECT 1 IS NOT DISTINCT FROM 1;
?column?
----------
t
(1 row)
postgres=# SELECT NULL IS NOT DISTINCT FROM NULL;
?column?
----------
t
(1 row)`ISNULL(NULLIF(A, B), NULLIF(B, A)) IS NULL`
What are you? human?
Truth table here: https://wiki.postgresql.org/wiki/Is_distinct_from#Truth_Tabl...
That bothers me. It is like saying that "infinity" (+inf) equals other infinity. Which is just wrong.
Some other SQL dialects will return "UNKNOWN" rather than "t" (true), which imho makes way more sense.
"SELECT NULL = NULL" will return NULL, use that if you want the standard way. There are many cases where you want it to return true though, and the boilerplate for that sucks, so IS NOT DISTINCT FROM has a very useful place in the dialect.
http://thoughts.davisjeff.com/2009/08/02/what-is-the-deal-wi...
"NULL semantics are designed to make you think you understand them, and think that the semantics are intuitive, and think that it’s part of some ingenious consistent system for managing missing information. But none of those things are true."
Any explanation you come up with does not generalize to other cases. NULL is just a bunch of special cases piled on top of each other, and really only kind of works because SQL is not designed to be very extensible (and to the extent it is, the standard doesn't give much guidance on how NULL should behave in these extensions).
Whether you consider that "good" or not is a matter of taste - personally I quite like it because it is very explicit and non-magical and I think NULL semantics cause more problems than they solve.
But that's just weird -- in a normal programming language you would never duplicate the tuples and make this mess; you would keep a single tuple on the LHS and group the RHS into a set (or array/list). Then you'd aggregate the set on the RHS. That model is simple because an empty set is a value with known semantics (sum of an empty set is zero), whereas a half-NULL row is just a hack.
Before someone asks, this has nothing to do with declarative/imperative, and nothing to do with some purity of the relational model. You can still have a declarative relational model with values that happen to be sets (or even relations).
Some programming languages have a rich set of containers to replace what SQL does with outer joins, others simply lack them, and pile hacks over hacks that are always much worse than the SQL one.
select school, (select array_agg(teacher) from teachers where teachers.school = schools.school) from schools;
If you don't like nulls any more than outer joins, you'd probably want: select school, (select coalesce(array_agg(teacher), '{}'::text[]) from teachers where teachers.school = schools.school) from schools;A "pure" relational model (like the one C.J. Date has been going on about for decades) has no outer join.
I'm not a hardcore DB person, but I've designed and modified schemas and queries for quite a few DBs over the years and not once can I recall feeling the need to do an outer join for production code, nor do I recall seeing a hardcore DB person using them (but that might just reflect limited exposure on my part).
On the other hand, I do see LEFT OUTER JOINs a lot in code written by inexperienced people.
Basically, the main use case I have for "dynamically creating" NULLs is quick inspection of data by shoe-horning something with more dimensions into the rows with equal numbers of columns model (CUBE and also OUTER JOIN) – but this is essentially just for display purposes.
From a data modelling perspective outer joins seem quite dodgy to me and an aggregating subquery (such as in the teachers example) is not really conceptually equivalent, even if you can implement them with left joins, and in some DBs such as MySQL they have been traditionally faster. An aggregating subquery is a perfectly conceptually "clean" operation, no ternary logic involved.
In a data-warehouse built up from event-sourced data, "missing" and "unknown" are the same thing: "I don't have this in here yet, so once I get it, it might turn out to be anything."
For a column with a value you know is "nothing", I'd suggest you keep a separate column to denote the type of the column, just as you would in a union data structure. After all, the explicit choice to have or not have a value is, itself, data, that you can either be in possession of, or not. You can't use one sentinel value to indicate both "absence of value" and "absence of knowledge of absence-state of value"; you need two! (Of course, if your knowledge of the absence-state is absent, the data is probably absent too, so you'd probably have a CHECK constraint to assert that if the tag is NULL, the data must also be NULL.)
I have a friend James, and another friend Jane. I don't know their ages. Are they of the same age?
Or Jane has a car. James has a blue car. Do car colours differ?
SQL provides human answers: explicitly answers "I don't know".
Other languages treat null as if it was just another value, providing very confusing answers. I wish all programming languages would use SQL style of null.
Even SQL databases that adhere to ANSI NULLS must treat NULL==NULL sometimes (group by, order by)
Obviously you need to compare nulls hence the crazy IS NOT DISTINCT FROM which is just another way of saying "equal" in English.
For example, if your items represent tags, and you're grouping by parent_id to find subtags, the ones without parents (parent_id IS NULL) aren't all subtags of some root of a tag tree; they're all distinct root tags in a tag forest.
Do I know the same amount about whether their age or their car colors differ? : Yes
And that is the crux of the intuitive breakdown of 3VL IMO.
"Is the answer to 'Are they of the same age?' the same as the answer to 'Do car colors differ?'"
Both answers are "I don't know". So... are they the same answer? The human in me says "Yes".
Of course you will recognize the similarity between the above and your first example. The interesting part is in the subtle difference between:
"Are they of the same age?" --> "I don't know"
and
2. "Is the answer to 'What is James' age?' the same as the answer to 'What is Jane's age?'" --> "Yes"
The answer to number 2 relies on the fact that I _know_ I don't know _either_ of their ages. This extra piece of knowledge allows me to "lift" null into a value rather than the absence of. It's the difference between:
`Maybe<T>.value == Maybe<T>.value`
and
`Maybe<T> == Maybe<T>`
You don't know the answer to the question "Are they the same age" (but it's either true of false), and you don't know the answer to the question "Do car colors differ" (it's either true or false). If you don't know the answer to either question then you don't know whether the answers differ either.
Even if they were, it's just a horrible name for the concept. You could argue that back when the keyword was introduced into SQL, it didn't yet have a well-established meaning that was different... but I'm not even sure that is true - some form of NIL or NULL has been around since the early Algol extensions, and it never meant "unknown".
The way to fix it would be to have MISSING and UNKNOWN as two different concepts. Then:
UNKNOWN + 1 -> UNKNOWN
MISSING + 1 -> *error*
TRUE AND UNKNOWN -> UNKNOWN
TRUE AND MISSING -> *error*
UNKNOWN = UNKNOWN -> UNKNOWN
MISSING = MISSING -> TRUE
Well, and fix the aggregates - e.g. SUM() of no values should just be 0.NULL doesn't really mean unknown; it means that instead of having a pointer to a result-set, you have no such pointer.
But in the semantics of answering questions, ala Prolog, "the absence of a result set" means that you don't know something, since those "results" are the answers to a question.
An alternative way to think of NULL is that it's a short-circuiting of what would otherwise be the database blocking on a promise for data to resolve, that never resolves—waiting for its deductive algorithm to get the data necessary to definitively answer the question.
> have MISSING and UNKNOWN as two different concepts
You already do. Just have a boolean column "foo_present" (or an enum column "foo_type", where one type means "intentionally absent") beside your nullable whatever-typed column "foo". In this case, a foo_present=TRUE together with foo=NULL means you're missing the foo value that you really expected to be there—it's a semantically-present, in-practice-unknown value (which probably indicates that e.g. the enrichment step hasn't gotten around to filling it out yet.) Meanwhile, if foo_present=FALSE, the value of foo itself doesn't matter (though it probably should be NULL) since you won't be looking at it in the first place; whatever value it has in practice, it's not meant to have a value semantically. It's pretty easy to construct queries around this paradigm.
Or, if you're using a SQL RDBMS that supports custom types, you can just declare a sum type, e.g. Pet = Present pet_id | Nothing.
Or, if you're using an SQL RDBMS that supports array product-types (e.g. "int[]"), you could just use an array-typed column with a CHECK constraint on it limiting it to have either 0 or 1 value. Then a value can be [], [foo], or NULL.
All of these allow you to see when the information of absence has been explicitly entered, separately from the query-response of unknown-ness which can come from a variety of places (such as e.g. joining several tables for sub-types of something together, where the columns that don't exist for a given sub-type will appear NULL.)
NULL in SQL has nothing to do with pointers.
It means that the value is missing or otherwise unknown.
Consider the column that is tracing the reading of a meter of some kind. And let's say that we could not get a reading -- due to some fault. So we did a reading, and we want to record that. What "value" should we record? 0 is no good - may make you think that we recorded a reading of 0 - which is not true.
NULL is the solution - it specifically tells you that the reading is missing or inapplicable.
Obviously, this complicates things, but incidentally translates well to other languages that have a (completely unrelated) concept of a null pointer.
Let me try another analogy, that might suit you better: if you think of each row-tuple in a relation in SQL as an asserted "fact" in a logic knowledge-base, e.g.
INSERT INTO parents VALUES ('x', 'y')
equates to the logical assertion: parents('x', 'y').
...then a NULL result in SQL is the equivalent of asking your theorem-prover to look at this knowledge-base and deduce the truth-state of other fact you never told it anything about, e.g.: ?- parents('a', 'b').
...and getting back "unsat" — meaning that the theorem-prover couldn't prove your assertion true or false.A theorem prover has the option to answer any question as Satisfied (Result) | Unsatisfied. This is what I meant by saying "a pointer to a result set, or a lack thereof." NULL is when you don't have such a pointer—which is why it's called NULL, by analogy to the DBMS's C implementation where the rowset pointer itself is likely set to be the pointer sentinel value ("NULL" or 0), rather than set to point to an empty rowset. The name NULL probably ended up in SQL after "bubbling up" from the implementations of the DBMS engines in this way.
The Weird Thing About SQL is that the grammar is phrased in terms of individual rows, whereas the query-planning occurs in terms of relations. So, when doing joins, you need a way to talk about what maps to "a join of a populated relation to an unpopulated relation", which in terms of individual rows, necessarily translates to "a real row-tuple value on one side of the join, along with some scalar sentinel row-tuple value [NULL] on the other side, standing in for 'a member of' the empty result-set."
That's the core semantics of NULL in SQL—to give you a 'scalar handle onto' the fact that there's no row on the other side of a join. That's where NULLs "come from", if you don't introduce them yourself (i.e. if you never make any of the columns in your tables nullable.) And so those semantics must be what you look at when figuring out what NULL "means." In joins, NULL doesn't quite mean "missing" or "unknown", but rather a combination of the two; it means "unsatisfied", just like in the theorem prover; it means that the query-planner has not yet been told enough to resolve your query fully.
(Yes, you can model an ingestion semantics where any row-tuple value that's not "in" a table, means that whatever predicate is represented by the presence of a value in the table should be considered to resolve false in the relation—but that's not really how SQL query-planning sees the world. It thinks of rows it doesn't have as "nobody bothered to tell me this yet"—unsatisfied—rather than "explicitly missing"—false. If you want false-ness semantics in SQL, it'd probably be better to define a view that generates rows with zero-values/false-values where the underlying table doesn't have values, and join to that instead.)
-- Find all customers with total orders
-- less than $100 in 2019
SELECT c.name
FROM customers c LEFT JOIN orders o
ON (c.id = o.customer_id)
WHERE DATE_TRUNC('year', o.ts) = '2019-01-01'::timestamptz
GROUP BY c.name
HAVING SUM(o.price) < 100;
Can you find the problem?[ SPOILER ]
The problem is if there is a customer with no orders. The left join will produce NULLs for the attributes of "orders", and SUM() will sum them up to NULL because it thinks NULL means "nothing". The predicate "NULL < 100" will then think the NULL means "unknown" and produce the result "unknown". Then, the HAVING clause treats NULL like FALSE, and does not produce the row.
WHERE EXTRACT(YEAR FROM o.ts) = 2019
...though I imagine some people dislike the way EXTRACT treats words like YEAR as keywords. Always interesting to see which solution people choose when there are several options. :) WHERE DATE_TRUNC('year', o.ts) = '2019-01-01'::timestamptz
will filter out customers without orders even before the HAVING (because DATE_TRUNC of NULL is NULL)? I mean, I guess you're still demonstrating the perils of NULL with that ;)If a JOIN could somehow return hierarchically structured data then there would've been no gotchas here.
But yes, I think left join is fundamentally about grouping, and grouping would be better solved with set-valued attributes or relation-valued attributes.
I wouldn't call that "hierarchically-structured data" because that has the connotation that the data is actually stored in a hierarchy, which I think is generally a bad idea (though not always).
A language should help you avoid making mistakes, not trick you into making them. The path of least resistance should be the safest and most normal thing, and SQL fails here.
But saying SQL is tricking people into making mistakes? Nah. The real issue seems to be writing a query without a proper understanding of NULL, not being tricked by the language.
So if you don't start out with any NULLs, and you have complete and unambiguous knowledge of your entire data set, how does "unknown" get injected into such an innocent-looking example?
So in your case I'd do
SELECT c.name
FROM customers c
WHERE EXISTS (
SELECT 1 FROM orders o
WHERE c.id = o.customer_id
AND DATE_TRUNC('year', o.ts) = '2019-01-01'::timestamptz
HAVING SUM(o.price) < 100
)
Again, might be dumb, but I usually find it easier to think in terms of subqueries rather than joins. Not that I find joining hard per se, just feels more natural and to the point to use subqueries. Database I use also seems to optimize it better most of the time.Could just modify it to exclude the orders above 100 ("where not exists"...), no?
Edit: just read what you said was wrong with it, and realized you missed the one in the where clause.
Like you said, really hammers home how hard to deal with null is with properly all the time.
It is actually the other way around. The bare numeral 0, in C, is defined to be the null pointer when the numeral 0 is encountered in a pointer context. This does not mean that the null pointer actually has anything to do with the value 0. The succeeding if clause in the C program in the article, the one which prints “NULL is 0”, is not even using any pointers! Since the C preprocessor is text-based, the token sequence “if(0 == NULL)” is replaced by the preprocessor with “if(0 == 0)”, which the compiler will interpret as an integer comparison, not a pointer comparison.
For further reading, see http://c-faq.com/null
And in general, "x <> 5" should be "true" if x is null, and not return null.
The justifications are mostly theoretical, not practical. I'd rather have 10 bugs from misuse of theory than 100 from real world. Theorists seem to magnify the "sin" value of theory-related bugs compared to actual bugs. I count them the SAME, and under that scoring formula, we are doing it wrong.
Configuration settings that change SQL semantics are just too dangerous.
From the official docs:
"Note that this option only affects the exact form = NULL, not other comparison operators or other expressions that are computationally equivalent to some expression involving the equals operator (such as IN). Thus, this option is not a general fix for bad programming."
https://www.postgresql.org/docs/12/runtime-config-compatible...
I know it seems tempting to use this setting as a convenience, but I strongly recommend against it.
To be pedantic, all that's evident from the observed behaviour is that in C, NULL compares equal to the integer literal 0. The expressions may be, and usually are not mutually interchangeable both ways, as they (may) have different types.
my experience: it's far better to explicitly reserve a value for NULL. In fact, I go the other way and ask when NULL is worth adding?
example: in your datatype NULL-values exist but there's no easy way to represent them otherwise, i.e. you can't reserve a value because they're truly all used by real data.
example: you have a slew of BOOLEAN columns and NULL is a legitimate third value in the dataset. The alternative (char(1)) can be a lot of overhead vs the bitmask optimizations that most SQL engines offer (but test this!!!)
hope this helps!
p.s. having studied the history and known some of the people responsible... if we gave them a time machine, they'd first kill Hitler... and then undo this decision.
It's a huge mistake in every language including SQL.
If you have an entity of a type. Say Int. What does NULL mean in context of that type?
X = NULL
Y = NULL + 1
It doesn't matter what language you operate in there is no NULL + 1, because NULL is not an INT. It just doesn't exist. So your programming language, if it's good, it should throw an error. If it's a shit it will continue to propogate the y deep into your program and you won't even know X was a null.If you think a default to zero works imagine this:
X = IntFunctionThatErrorsOutAndReturnsUnknownValueAkaNull()
Y = X + 3
Does it make any sense to you that Y will be 3? X was unknown! You don't even know that the function errored out! Either IntFunctionThatErrorsOutAndReturnsUnknownValueAkaNull should have thrown an error or it should have returned a special type called an Optional Type. Throwing an error isn't ideal because you won't know something is wrong until runtime so the Optional Type is the better deal here. Either way neither approach is taken and instead you have a null and Y propagates through your program as if NOTHING happened.Unfortunately SQL doesn't offer The Optional type or sum types therefore we are stuck with Null.
So if you do this in a language that defaults nulls to zeros:
x = getFirstSqlColumnAsList("SELECT * FROM integerTableWithOnlyNullValues;")
y = x[0] + 1
boom you get y a garbage value propagating through your entire program thanks to SQL nulls. SQL isn't innocent here.Now imagine Optional types with exhaustive pattern matching:
x: Maybe(int) = getFirstSqlColumnAsList("SELECT * FROM integerTableWithOnlyNullValues;")
y = x match {
Just x: Just(x[0] + 1),
Nothing: Nothing
}
If you can't understand why the above code is so much better than the previous code than you need to look up the Maybe Monad with algebraic data types. Basically the code above makes sure that there is Zero chance that an error is unhandled.