So, how is this approach significantly better than good database design and practices ensuring data consistency? E.g. column length, foreign keys, stored procedures for updates/inserts that require validation logic?
> Some databases can enforce relationships at the systems level, using foreign keys, constraints, and other similar concepts, but these built-in checks and constraints represent only a small set of possible characteristics that may be modeled in any given system. Likewise, ORMs like ActiveRecord can check data at write time using validations, but these often need to be disabled in performance sensitive code, and can themselves contain bugs.
For example, I built similar tool that you could write simple rules to validate product data. This would run every night and generate a report for the data quality team. One of the advantages was that you could decide have stricter requirements and then choose if they only applied to new products, or if the team had to go back and fix all the old products too.
Some rules I remember:
- Validating min-length, max-length, and casing of product names. Spell checking words founds.
- Verifying product images existed and where the right dimensions, below a certain file size, etc.
- Making sure that certain product categories had dimensions, and that the metric/imperial conversions were rounded in the industry-norm way (e.g. 3 3/4" was 96mm)
- Making sure that pricing/margins made sense on a vendor/product category level, to try to catch typos like $1 or $1000.
The thing I found handy was to add new rules when problems occurred that fixed historical issues.
Everyone introduces a cache (aka eventual consistency) at some point. Everyone ships a bug here and there. Everyone messes up a rollout eventually.
Just like you should have backups even if you never intend to DROP TABLE *, you should also be confident that the invariants you design your system around are actually enforced.