Comment by protomyth | Hacker News Reader

Comment by protomyth | Hacker News Reader

A lot of the problems I have seen with queries (other than DBA issues) is the conflict between application developers and report writers. A lot of databases are designed for transactions and resources are not often available to do a proper reporting database or at least summary data.

I have a very simple rule for myself - "if a user of the application is concerned about a certain attribute or state an element (e.g. person, truck, plane) is in, then a report will be required showing all elements with that attribute or state."

If your database design cannot support that rule, then trouble will happen and you will have serious performance problems.

To give a simple example, suppose you are running a group of storage garages. You have a table with all your customers, a table with all your storage units, an assoc table joining customer and units with active flag + date of start, and a table with all your payments. Good enough to do transactions and figure out for a unit if they are payed up.

On the other hand, writing the report to tell who hasn't paid is going to be kind of a pain. It is a simple example, but not much different from what you find in large systems.

Is your point here that the data being stored is insufficient (e.g. you'd want an end date, not just an active flag; this doesn't cope at all with prices changing over time, bulk discounts, or different customers paying different rates; there's no concept of invoices, or whether payment is due based on calendar months or based on opening date; etc) or that you're ignoring all that sort of stuff just to keep the example simple (so assume everyone pays a fixed rate per unit, due weekly; someone can't close their account until they're paid up; etc) but that you'd still want a more complex schema so as to be able to more easily generate a "Who owes us money?" report?

If it's the former, then sure: you need to be able to model all these things properly. If it's the latter, then I'm not so sure. The SQL to create that sort of report is going to be non-trivial, but it shouldn't be overly complex for someone who knows what they're doing, and if you have the correct indexes it shouldn't take very long to run either.

If you want to start doing all sorts of fancy data warehouse slicing and dicing, you're usually better extracting daily (or more/less frequent depending on needs) dumps of your transactional database into a different structure more suitable for reporting, than in restructuring your 'live' database and having to deal with all the resulting denormalisation issues, etc.

As stated, I believe all the information needed to generate the report is there. My point was that what is sufficient for an application is generally insufficient for basic reporting. So, when budgets get short, the database work to make queries by reporting easier is often ignored and complicated queries and processes become the norm. Those laying the problem solely at the feet of the DBA's are missing the other group that tends to make these queries necessary: App Developers. I once had an app developer tell me one type of invoice was impossible to make because some of the relationships and data were intrinsic to the application and would need to be modeled the same as the application. He wasn't far off.

There is a point between transactional and data warehousing that needs to be hit. Simple summaries or considering the question of "how do I retrieve everything in this state" will sometimes suffice. Building a system to get single transactions in an out tends to make routine report take all night or be impossible to get in a reasonable amount of time.

If your schema and app requires non-trivial SQL, then expect maintenance nightmares and lack of ability to scale.

Not always. An app requiring non-trivial SQL can be an indication of a problem requiring non-trivial workflows.

If the problem is complex, you want to model relations, not ignore them.

(But clearly not in this hairy query.)

The denormalization argument is misguided.

Denormalization is a performance consideration, and now even happens transparently at the storage layer in some RDBMs. So restructuring a 'live' database should be done for semantic reasons: because the workflow changes; because the app needs to change.

Reply on news.ycombinator.com