Using the Specification Pattern to Build a Data-Driven Rules Engine

Real business rules engines use something like

https://en.wikipedia.org/wiki/Rete_algorithm

Production rules systems in the 1970s (when they were in vogue for "expert systems") did not use indexes and generally had poor execution performance compared to modern "business rules" engine.

Production rules are related to other logic-based methods of programming such as "logic programming" (which tends to progress backwards from a desired consequence to prove it is satisfied) or "theorem provers" that could demonstrate that a section of code satisfies certain invariants.

> You can sort sub-clauses in an AndSpecification so that the ones most likely to be false are first, so that you are more likely to short-circuit and cut down on evaluation time (and you sort in the reverse order for the OrSpecification).

Similar trick also works in context of Alpha beta pruning when minimax searching through a game tree: at each node various moves are available, getting tighter Alpha beta bounds for pruning earlier in the search gives a massive speedup, getting tighter bounds corresponds to selecting a relatively strong move to investigate first. So if you have some heuristic to guess what a good move is for the current player and use that heuristic to define the order that moves are attempted, that can give a large speedup.

Unlike some other situations with heuristics (e.g. a*, where heuristic estimating distance to goal must always be a lower bound to true difference to goal in order to guarantee that the solution obtained by the search is optimal) , there's no requirement that the heuristic used to order moves during search have any guaranteed relationship to the true order -- so that frees you to use wacky empirical heuristics -- e.g. fitting a statistical machine learning model to suggest the search order based on features that can be defined and quickly evaluated at each search node.

When I played around with this idea to accelerate minimax search it generated a fun little project to instrument the search with metrics to collect training data, fit a simple model to predict a strong move (i used random forest) and then make the model evaluation run really fast at predict time: the model was fitted offline before search so I could encode the fitted model -- a bunch of decision trees -- into C code (lots of gotos!) that was then included into the game search code

Data-driven approaches leave a lot implicit, meaning that it can be hard to answer the question, "What's really possible here?" And they can only be verified to the extent you have all the data, which usually means a lot of checks are deferred to run time.

I can certainly understand using something like this when your users are building rules, because you don't know what they want to build so you just provide the base implementations and the operators. But if engineers are implementing all the logic, it seems like the criticisms called out in Wikipedia regarding obfuscation and re-implementation of basic operators would apply.

https://en.wikipedia.org/wiki/Specification_pattern#Criticis...

From Common Lisp's perspective, the example given is just:

  `(and (> divident 0.025)
        (< payout-ratio 0.5)
        (or (< pe-ratio 20)
            (< price 130)))

Or, you can wrap it in a (lambda (divident payout-ratio pe-ratio price) ...) and pass to (compile ...), and get a piece of executable code. With perhaps a line or two of book-keeping (that can be added by your code, and not necessarily typed in by the user), you can make the rule declare which variables it cares about. So a full example:

  (let* ((rule '(lambda (data)
                  (destructuring-bind (&key divident payout-ratio pe-ratio price &allow-other-keys) data
                    (and (> divident 0.025)
                         (< payout-ratio 0.5)
                         (or (< pe-ratio 20)
                             (< price 130))))))
         (executable-rule (compile nil rule)))
    (list (funcall executable-rule '(:divident 0.5 :payout-ratio 0.4 :pe-ratio 50 :price 10 :other-data 42))
          (funcall executable-rule '(:divident 0.5 :payout-ratio 0.5 :pe-ratio 50 :price 10 :other-data 42))))
  
  ;; Result:
  (T NIL)