Using “Git for data” to find and analyze test regressions

Blog author here. This post discusses how we use our git for data product, Dolt, to store test results, and how this lets us use SQL queries to analyze test regressions on our 6M+ test query dataset. Example:

  dolt_sql_performance> select nightly.query_string, nightly.error_message 
                     -> from nightly_dolt_results nightly join releases_dolt_results release 
                     -> on nightly.test_file = release.test_file and nightly.line_num = releases.line_num 
                     -> where nightly.result = 'not ok' and release.result='ok' limit 1;
  +-------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+
  | query_string                                                                                                                  | error_message                         |
  +-------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+
  | SELECT * FROM tab0 WHERE - + 63 IN ( col3, - col3 * col1 + + ( + + col4 ), 67 * col3 + - + CAST( - col1 AS DECIMAL ) + col0 ) | Unexpected error invalid type: string |
  +-------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+

hm. pushing from the test runners to a central place would make sense for that, and could be quite interesting to keep track of some things, but it doesn't seem like there's a viable way of doing that internally (=not relying on some external service) yet?

CEO of company that built this here. You need a versioned SQL database in your toolbox.