Hacker News Reader

Comment by RedShift1 | Hacker News Reader

When I started a project with SQLite, the available data types struck me as odd (coming from MySQL), but as I learned I started asking the question: are there any more fundamental data types other than null, int, real, text and blob? For example dates are just a facade for an integer of some kind, JSON is really just text adhering to certain formatting rules, booleans are usually stored as some kind of byte anyway so why not drop that abstraction?

With this limited set of datatypes it really makes you think harder about the data you are processing, because in the end all your data is one of these types anyway.

View on HN

> are there any more fundamental data types other than null, int, real, text and blob?

Those type names are just hints, they don't constrain the valid values in any way.

https://dbfiddle.uk/?rdbms=sqlite_3.27&fiddle=4634e3821676ed...

I can't believe that runs. What a misfeature.

You just put me off Sqlite for life. I can't believe someone thought this to be a good idea.

SQLite just does what JavaScript and other weakly typed languages do. The developer is responsible for what they input. If type checks are a must, SQLite is just not for you, easy as that.

But event JavaScript will reject data if it's considered not valid for a type.

It won't throw but it will set an "invalid" / "NaN" / etc. value.

I don't see how having "NaN" or "undefined" in a number field makes any difference to just putting the value there. If you are using a strongly typed language, this will not be a problem.

SQLite made a design choice in favor of simplicity. It's also missing basic date functions all together. The only way to compare dates is by using Unix Epoch.

I use ISO 8601 full timestamps in UTC.

They were carefully designed so that collation order is identical to temporal order. Which is convenient!

If you need interval logic, though, SQLite won't help you, and epoch is the better choice. It's possible to solve some queries with a regex, but you won't love it.

Chill out. This is a design choice to make it more viable to be used in dynamically typed languages. It's maybe not the design choice everyone would make, but it's not, like, insane: this is how all variables in Python/JavaScript/Ruby/whatever works.

If you really care about this, adding strongly typed columns is trivial: https://dbfiddle.uk/?rdbms=sqlite_3.27&fiddle=9baffa184672a7...

That is... interesting...

It's a feature:

https://stackoverflow.com/questions/29476818/how-to-avoid-in...

I don't really see it as such a problem. When interacting with the database you either use an ORM with your strongly-typed business objects or you write the SQL and use parameters (again strongly typed)

SQL parameters aren't strongly typed though and in any case SQLLite doesn't have stored procedures.

> When interacting with the database you either use an ORM with your strongly-typed business objects

If you have strongly-type business objects then why not have a strongly typed storage? If you code can control the constraints that a correct data type ensures, then why have "strongly type business objects" to begin with? Why not store everything in your code as strings as well?

I see this misconception all the time. The database (and its data) lives way longer than most applications. And it's also a wrong to assume that there is always only one application accessing the database. Bulk loads are a typical case of secondary applications.

Not choosing the proper data type in a relational database is a really bad decision and we see question on stackoverflow and similar sites on a weekly (if not daily) basis asking how to fix invalid data in those "un-typed" columns.

I'm not saying it's useless to have type constraints in a database. Just that with a bit of care it's not such a big deal.

In the end there will always be some business rules that are not constrained by the database. So, you always have to be a bit careful about what you store in it. Indeed not being careful and hoping that your types, constraints and triggers are going to save you is more risky

> Not choosing the proper data type in a relational database is a really bad decision

Well, then rejoice, you can't make this bad decision in SQLite because everything is +/- a number or a string

I mean, you can represent everything under the sun as a string (or a very big int) but that's a level of operating where I find my brain starts to get fatigued for absolutely no real gain at all.

We need to store weights in kg with three decimal places. If we sum to get a total, the total should be correct according to those three digits.

I'm sure we could get it to work in SQLite, but it sounds like we would have to have some layer of manual fudging that we couldn't forget about.

In our current database we just use "numeric(16, 3)" and no worries.

You could store it in grams as an integer.

At the risk of sounding glib, you could store everything as integer grams?

Yes of course. That's why I said I'm sure we could get it to work. But then it has to be converted to kg for actual usage. Either directly in the queries or views.

Having a fixed-point decimal type allows us to not think about once the table is created.

Same issue with money. Most of the time we need to store monetary values with two decimal places (cents).

What kind of application are you developing where there is so little middleware between the database storage layer and the user that converting to/from kilos adds significant additional complexity, but your users aren't technical enough to just deal with having weights in grams?

These are electronic versions of official forms. The user interface and reports should show what is actually being sent or has been sent.

Sure we could deal with it, but there are quite a number of tables due to different forms and messages, and then there's all the reports. Many custom ones thanks to to local officials wanting data from a certain customer in a certain way...

Bureaucratic forms have a tendency to change unpredictably. Today they want weight in kilograms, tomorrow they'll want it in grams, and the day after that in standardized intergalactic weight units. I get what you're saying (and I don't like writing unnecessary mapping boilerplate too), but there should be a middle layer to protect the application from craziness like that.

Thankfully things in our sector is fairly static in that regard, so this is seldom an issue. That sounds quite horrible indeed.

And yes, a fatter middle layer would be nice. Our next generation software will probably have more of that, this code base is over 20 years old at this point...

I realize this response turned out a lot more harsh than I intended. I was in a slight rush so didn't formulate myself well, didn't mean to be abrasive.

> Most of the time we need to store monetary values with two decimal places (cents).

Yeah who cares about fractions of pennies anyway. Just makes things complicated.

Yeah for being so uptight about just about everything, the officials seems quite relaxed about values slipping between the cracks due to lack of decimal places.

Like, total invoice value can only be specified with two decimal digits, typically in foreign currency. Yet we also have to specify per-line value in local currency, also with only two digits. And then the per-line values are used to calculate taxes and whatnot...

The US IRS (tax agency) allows you to round nearly all values to dollars ignoring cents altogether.

With this sort of perspective, why use a database at all? You can store literally any digital object as some sequence of bytes, and that's what file systems are for.

At some point, you actually like for the software you use to actually have meaningful features.

Because sqlite let's you index and query the data without having to write your own layer to do that.

Having rich data types allows a lot of things without having to write your own layer. Spatial indexes are extremely efficient, and I can't query geometries efficiently when I'm storing geometries as a blob. The same goes for dates, json, XML, ranges, etc.

The problem with reserving specialized logic for the application layer is that it limits you to simplistic indexing schemes and you end up doing excessive IO and filtering in memory to get what you actually want.

The idea that databases shouldn't have specialized datatypes is really only an idea that works in simplistic crud apps. The world is much bigger than that.

Querying filesystems is extremely primitive when compared to what sqlite can do.

So is storing dates as integers, and json as strings.

But dates are integers (all databases store dates as an integer that is an offset from some fixed date), and JSONs are strings. SQLite has the functions to work with these datatypes (dates, JSON, etc...) but fundamentally they are not different from integers or strings.

And integers are bytes and strings are bytes and tables are bytes and indexes are bytes. If you're not happy with databases being intelligent, why stop with integers and strings, when you can do everything you want with bytes?! Memcached, or LevelDB, or even a file system is the ultimate database if you follow your arguments to their logical conclusion.

But data representation isn't data. You can't look at a serialized sequence of bytes and know what it represents without context - without a serialization scheme. For relational databases, the table definition is the serialization scheme - it is the context. For example, you can't know what date an integer is representing without contextual information like the offset. By storing dates as dates in your database, that context is baked in.

It is helpful to have that full context inside your database because it allows you to operate that database more efficiently (by carefully indexing on the properties of the data, not the data representation), but it also allows you to use that database in ways that are not tightly coupled to your application, such as analytics, because you are storing data itself and not just the bare minimum required to represent that data.

Reply on news.ycombinator.com