It would be very enjoyable to talk further about this. One reason I am hesitant to invest too much time into a prediction-focused side project is that I believe many organizations that purportedly use machine learning actually do not care about prediction, as opposed to marketing or demoware.
A major influence on me is a 2011 Cato Unbound article called "Who Cares About Forecast Accuracy" by Robin Hanson [0].
I actually only found the article a few months ago, but it crystallizes the experiences I've lived through for the past 5 years perfectly.
Since the social status concerns of executives are typically the primary concerns within any organization, even if a better forecast can add value in some ways, if it risks taking value away in the social status realm, everyone will veto it.
As a result, people are often hired purely for credential (this has happened to me before). Then you are asked to do things that are directly damaging to your own career because they take you so far away from your area of expertise or your specialization that they make it harder for you to be hired by anyone else.
I have not yet encountered any organization which actually engages in fundamental research about improving forecast accuracy or truly solving problems with A.I. or machine learning.
Instead, I've only encountered places that put a thick wrapper of marketing around a thin kernel of simplistic technology, and then spend all their efforts on scaling up the marketing and sales apparatuses around the fundamentally-simple thing. (Simple things are often perfectly fine, of course. But in these situations, the simplicity was not chosen because it was a cost-effective way to meet business needs. Rather, it was chosen because it was cheap and easier to politically control. And after making the choice, no further effort was ever spent investigating whether any type of alternative could improve upon the simple thing in a cost-saving way.)
For a long time, I was very bitter about this. But then I realized that if this is the only mode of employment for machine learning researchers and engineers, it might still be OK. If the other aspects of the job afford you ways of meeting other goals, then perhaps it's OK to sacrifice the aspect of intellectual curiosity.
Unfortunately for me, the other goals I have include having adequate privacy at work, focusing on software craftsmanship and being proud of things that I make professionally, and maintaining a healthy work/life balance. And it seems these things, like intellectual curiosity, are also mostly impossible to find in modern technology jobs.
At any rate, I am not convinced that it would be a wise investment of time to conduct this type of side project. One bad outcome would be for it to embody sort of cargo-cult data journalism practices, like mining for t-stats or something and milking a misrepresentative headline for a bunch of blog hits. An even worse outcome would be to actually solve a problem really well, parlay it into a job offer, and find you have been hired purely as office decoration, to produce demoware or increase value for a potential acqui-hire or whatever.
[0] http://www.cato-unbound.org/2011/07/13/robin-hanson/who-care...