So the system is built around the ~50 different indicators and oscillators that I've found the formulas for generating.
These "methods" are variable driven, meaning each method could have 1 to 4 variables ranging between 0.0001 to 260.
The current testing platform that it self manages has the end goal to find the highest gain after commissions using a combination of any of these methods for opens and closes.
Considering the __massive__ amount of possible combinations of these methods, the system has two testing suites it runs, the first round of testing is a simple test suite that only sums the gain it would get using those methods over the last YTD. This doesn't mean that the highest here will be the best combination in the end as commissions will bite you if you don't keep an eye on them. We simply only collect sum of gains because of the MASSIVE amount of tests we have to run. This cuts down on days/weeks/months of computation time.
So it will increment through the blocks (two sets of combinations, one for open, one for close) of functions typically by 1000, so it will start testing at #1, then skip 1000 of them, test #1001, rinse and repeat till its ran through the whole list (this is broken out across 5 machines around my condo, and multi threaded to utilize every core minus 1 of each machine, I have roughly 40 cores working on this system).
Once it has identified the best ~25% of raw gain increment points, it will then start incrementing forward and backward around those starting points as long as the raw gain is increasing and not showing a change in direction (I use a mixture of a momentum formula and P SAR to determine the change in direction as you will get some noise and a quick change could make you miss gold on the other side because of a slight fluctuation). So while all these are being ran, the results get funneled back into the job queue for the second half of the testing suite.
This is also distributed across all the machines.
The second half then runs the full test suite on each combination of methods to recreate the market verbatim for the last YTD and determine everything, gains, share quantity, p/l, commissions spent, highest equity, ect.
Finally all this data is put back into the job queue once again to be sorted to find the highest net gains after commissions.
The winning methods are finally stored in the datastore to be accessed by the actual stock trading platform that will use this data during trading hours to execute trades accordingly.
So from start to finish the application handles and manages, what tests it wants to run, can determine the best strategy to use and execute the strategies on its own.
I tried to keep it very basic explanation, there is a lot of other things that go on, to make this all work flawlessly. I do want to thank 0MQ and Gearman for playing vital roles in work distribution and message queuing amongst the worker threads.
I do highly recommend that if anyone every wants to truly learn how to scale an application to build a algo trading system on limited hardware and try to squeeze EVERY millisecond you can out of it. (I've rebuilt it from the ground up numerous times).
Actually typing this out makes me see some similarities to a map and reduce method in some ways as well.