Do you make these 5 database design mistakes? | Hacker News Reader

Do you make these 5 database design mistakes?

thomaslarock.com

54 pointsSQLRockstar37 comments

I think more important than obsession over column sizes (in most databases, varchar columns only use the space necessary, not the maximum, more or less) is using correct datatypes. E.g. using varchar for a column that you know will only have numbers. Or using int for timestamps (e.g. using the unix epoch offset) and then having to do a lot of converting. Or sticking serialized objects from your application domain into blob columns.

Most databases have a lot of built in safety and functionality if you are using the correct datatypes, but that is something you can't benefit from if you are treating everything like a blob or a string.

I've got to disagree about not using varchar for a column you you know will only contain numbers. I'm assuming you mean something like the numbers of a phone number (dashes and parens have been scrubbed out). Even if a field will only contain numbers I won't use a numeric based field type unless there is the possibility that I would be using the value in some form of mathematical function. I might add and subtract from an account balance but I would never add or subtract from a Social Security Number or zip code.

(I don't keep to this rule for surrogate keys which will be int/bigint auto incrementing, even though I won't be doing any math on the values themselves)

There is value in what your saying.

But, if you store a SSN as say 123-45-6789 your using at least 11 bytes to store a 4 byte number. Use it as an index and you waste that space again, store it in a cache and you have less space for useful information, compare it with another SSN and it's slower etc. Now individually it's probably irrelevant but start doing this with several fields on a large application and you can noticeably slow things down even if the extra disk space seems cheap.

PS: 95% of the time it's probably the safest option, but it's still something you should consider based on how that data fit's in with the rest of your application.

Number types are meant to store numbers, not text. Even if you know the text will be just a subset of the alphabet (just 0-9) it is not a number.

Just a couple of problems: some day in the future the field could need another character beyond 0-9. In Spain National Identity Numbers were once just numbers, but they changed and added a letter for validation. If you had stored this as a number you would need to change data types, and suffer a massive application update.

Another problem: if you order a number field you get something like 1, 5, 100, 2000. But if you order a text field you would get 1, 100, 2000, 5. It's not nice to get the "number" order when you are expecting a "text" order.

Some numeric data types will trim off leading 0's so if you have a number like 00543 when stored in the DB it will be retrieved as 543. Things like social security numbers and phones numbers are unique identifiers that just so happen to be numeric they should be treated as such with regards to data considerations.

Phone numbers are not unique identifiers. In several regions around the world, phone numbers are reused. Please keep this in mind because it is very short sighted to say you will never expand globally.

I agree, neither phone numbers nor social insurance numbers are unique.

But the issue is with storing them in numeric datatypes. They aren't guaranteed to be unique forever (and indeed, are alphanumeric in many jurisdictions) and the leading zero problem is a killer when you go to reconstitute them, both for performance and for logic reasons. Same goes for US ZipCodes.

This is why I have to have a very strong technical reason to make a non-math column numeric, especially with externally set data (like SSNs, SINs, Account codes, etc.) The people who set them could just start adding letters or symbols...and this isn't rare.

Indeed, Medicare numbers are usually the SSN with a letter appended

Sure perhaps I should not have used the term "unique" but they are identifiers, they identify how to contact a certain individual based on a numeric sequence. Same with SS numbers they identify a living person. I would never use either as a unique key in my database but I am not a natural key person for the most part. I tend to like a uniquely generated key to serve as a surrogate for my unique primary identifier.

Being cognizant of storage and indexing implications is important, but I would still probably go the char/varchar route; a distinction should be made between numeric data and text data that happens to be comprised only of numeric values. In addition whether talking about SSNs, product numbers, whatever, if you try and use a numeric field you run the risk of coming across a value that exceeds the upper bound of numeric type.

>> But, if you store a SSN as say 123-45-6789 your using at least 11 bytes to store a 4 byte number.

Plus it gets worse because people inevitably end up writing queries that parse the number with dashes, brackets, or other special delimiters. Those queries are CPU-intensive to do on the fly, and with databases like Oracle, DB2, and SQL Server charging by the CPU core, it becomes an expensive proposition as well.

Rather than the safest option "95% of the time", I'd say doing so is probably best "essentially all of the time except for a few unique circumstances, all of which you'll know when they happen."

"using varchar for a column that you know will only have numbers"

I've been bitten too many times by the recommendation of using numeric types for [0-9]+ data. Unless it's fundamentally going to involve arithmetic I say leave it as some kind of string type.

A rule of thumb: if you ever end up with a situation where you want to display data that is essentially just a table and you can't drive that via a fairly simple sql query then that probably means you need to revisit your design.

One mistake I used to see a lot is not paying attention to indices. It's sort of like the "go big" error in that you just create indices all over the place even if they could never be used. Also, it could be adding columns to the index that won't improve access time, or not paying attention to the order of columns.

In reality, you need to analyse your code and profile your application to see what actually is needed for your indices. Also essential is understanding the overhead that an index creates.

I don't see this as often today, however. I think that in a lot of cases developers put off creating indices of any sort until a performance problem materializes.

I think that in a lot of cases developers put off creating indices of any sort until a performance problem materializes.

Which is perfectly fine.

Not perfectly fine if you use MySQL. Adding an index to a table on MySQL locks the entire table, which can introduce hours of downtime if the size of the table is large.

The problem there is two-fold:

1. Not having done proper index analysis in the first place. While you can't get it right 100% of the time if you are expecting tables to grow that large you really should think hard about thsi sort of thing as close to the start as practical.

2. Using a database system that can't perform an online index build without locking the whole table. I know that such an operation needs to aquire some locks during its activity no matter what system you use, which will create performance issues for your live site if you are not able to schedule the index change in a pre-planned "maintainence" downtime (i.e. an application that has significant "high availability" requirements), but requiring a full table lock here seems to be a fault in a system that claims to be "enterprise ready".

Exactly. There's a difference between "putting off indexes" and "waiting until it takes hours to add them." The problem is likely evident well before that point.

Agreed. And certainly better than creating bad indices.

I wonder: are there any good rules-of-thumb for when to go ahead and add indices at table-creation time?

Add an index to support foreign key relationships. If you've got a SalesHeader table and a SalesDetail table, you want an index on SalesDetail.SalesHeaderID, especially if you allow cascading deletes from the SalesHeader table.

Projected usage. It can be done, but you have to be an expert basically.

On "not going big, just in case" I don't disagree, but the body text implies that excessive use of storage space is the problem you are creating for yourself, which isn't the most impoartant factor here by quite a margin.

Space is cheap. What aren't as cheap are memory and I/O bandwidth: using large datatypes limits the size of working-set you can fit into a given amount of memory, and slows down the process of reading data from permenant storage into memory when needed and not already present.

Increasing the load on your I/O capability in this way is far more of a problem than the extra storage space consumed.

I've got to push back on the anti-guid stance for surrogate keys. They may be big but it is nice to have opaque IDs so that no one makes flawed assumptions about ordering or relative insertion time/trying to guess valid IDs, etc. Opaque is good...

The size of guids isn't the only challenge. (The size is a little trickier than it looks, too - remember that the guid will be included in every index so the database can find its way back to the clustered index.)

The other problem with guids is that if you cluster on a guid, then whenever you do inserts, you're inserting the data into random spots in the table. You end up fragmenting the bejeezus out of the table, doing page splits like crazy. If you defrag the table/indexes, you'll be right back where you started within a few days of doing inserts.

That penalty isn't immediately obvious in small environments, but by the time you're big enough to have performance problems and you call in a consultant, it's going to be an ugly discussion with management. "Hey, wish I could help you, but..." Don't get me wrong, that's not always the biggest bottleneck, but it makes for some awkward discussions.

GUIDs are HUGE and have a significant impact on performance. Go read Kimberly Tripp's blog post referenced in the article. She does the math for you.

In almost all the cases that I see GUIDs, they were totally unnecessary for the design. Even the developer who designed them could not give a reason why they needed to be GUIDs. A row unique across the entire universe? Really?

I'm not saying there are no cases...just that in most cases they negatively impact performance with little business or logic gain for that price. All design decisions come down to cost, benefit and risk.

How do modern distributed OLTP systems deal with generating unique sequence numbers? Back in the day, this was a big problem, and I always thought that GUIDs would someday be a solution (though at the time, any string was too big/slow to be a primary key). Having one key-issuing server was a SPF; sharding the key ranges by server made it difficult to add new servers.

You can have the main key issuing server issue blocks to distributed key issuing servers. The keys are not necessarily increasing with respect to row creation time nor are they gapless but it's good enough for many purposes. It's also still a SPoF but not a single point of contention bad enough to affect performance significantly.

Interesting.. how do you deal with ID collisions after rollover? That was our big problem, even with 8-byte IDs - the ID allocator had to know the Global Truth. Dealing with collisions as "exception, try the next one" was too expensive, though with faster hardware that may be moot.

Yes, rollover is an interesting problem. With the app and scale that we're operating at that will not happen within the next few thousand centuries though. I'm actually describing how I might go about doing it, and what I /think/ Oracle RAC does.

I'm not actually certain as to the implementation.

This was my sentiment exactly. Any idea why he's so keen on sequential keys? They're mentioned as desirable several times in the article.

The issue, particularly with innodb on MySQL is that the row data is stored with the primary key. By using GUID's and not sequential ids, you end up having to rearrange the entire dataset to insert the row data at the appropiate place, to keep the index in order, instead of appending it, killing your write performance.

I tend to view designing database schemas as an evolving process. Picking larger datatypes to me is sometimes easier than having to change it at a later date. If space really does become an issue I can deal with it later.

You need to really understand the data you plan to put in.

This webpage causes my chrome to lock up and my laptop fan to go on full strength. Heavy HEAVY social media and user tracking javascript.

I never took databases on college, don't have energy to learn about normal forms, but find "designing" the schemas for my simple services a joy.

I do see that you can go a long way with relational databases just using experience.

However, they do have a formalised mathematical basis and I think you dismiss it too quickly. If you do spend the time to learn the maths then it will push your abilities that little bit further. You'll also likely find it quite easy to pick up given that you already understand the field well.

Reply on news.ycombinator.com