Experimenting with a Neural Network-based Poker Bot
mattmazur.com
mattmazur.com
A short stack raising >11% when he shouldn't be is a pretty severe leak ;)
Just a thought: I don't see the implementation of the algorithm here, but if you're using the values of individual cards scaled to 1, then you are saying that v(A) - v(K) = v(8) - v(7) which seems absurd. Imagine that there were (fantasize for the sake of argument) a few shoves in front of you, then should be that v(AK) - v(KQ) >> v(87) - v(76).
I'd need to give more thought to this problem in general, but I have to ask Matt...why did you choose short stacking full ring NLHE? It seems mechanical for someone very familiar with the game, but it's so situational. The first area I'd think you'd be most effective would be LHE-HU-SNG's.
Anyway, I think I was thinking that including the difference from the average value would help the neural network identify pairs. In retrospect, it should have been able to recognize that simply from the card values, no?
Limit holdem had been done a lot and No Limit was my area of expertise, which is why I chose it. I didn't have that much experience with shortstacking, but I had read pretty much everything there was to read on it and it seemed like a good start because of its simplicity. Little did I know that the profits from shortstacking come from a very few carefully timed moves that are close to impossible for a rules-based bot to emulate (more on that in another post). Eventually I decided to build a no limit heads up bot, which is where my success wound up being.
I didn't mention it in the post, but the key was moving away from rules based to a value based system, where the bot calculated the profitability of its options and decided based on that. It was a lot easier to debug, but there were a whole set of challenges to that too, like estimating your opponent's range.
Instantiating the features correctly is crucial to getting good neural network accuracy.
It might be the case that a thermometer representation for the card value is more appropriate. The thermometer representation is that there are thirteen bit-values, and the first n values are on for the value n.
Using this sort of input representation, the network can model the non-linear relationship between different card values, instead of assuming a strictly linear correlation.
In particular, read the paper titled "Opponent Modeling in Poker: Learning and Acting in a Hostile Environment". Section 4 outlines several prediction methods, one of which is a neural net.
2) A site that a human player plays at against the computer. The computer also tries to model the player using a NN and tell the player how "Big" of a NN was needed to model their play. Perhaps how quickly the computer was able to start predicting the player's play? The idea being that you want your play to be very hard to model with a NN so that you are more unpredictable in real life.
Note that this is all based on personal experiece, and may not be the common case, but I feel like the average hacker will play to test the bot, not to win.
The real challenge is finding enough tables to play. The bad thing about short stack is that sometimes someone calls you and you win, and now you've got at least a medium stack. Most sites don't let you come back to a no-limit table with less money than you left with until some period of time (for instance 2 hours) has passed. Party Poker did not have this feature (maybe still doesn't) so it would have been easy back then, but for an American now I don't know of any site of substantial volume where you wouldn't quickly burn through all available tables and find yourself waiting.
Even for a bot, there might not be much profit in it.
I purchased 7M+ NL200 hand histories from a guy who mined them all day every day. I filtered that down to hands where someone was playing with 15 - 25BB stacks and then for every person that played more than 5,000 hands in that range, calculated their profits and winrates.
Here's the breakdown: http://www.mattmazur.com/images/ShortstackingNL200.png
It's important to remember when looking at these numbers that the vast majority of people who try shortstacking never make it to 5,000 hands so this is heavily skewed toward the survivors. It would be like analyzing the revenue of startups that are over five years only and then using that average for the average revenue of all startups. There's only 80 people in the full ring group and 19 in the 6-max group. Just looking at these folks, only about 1 in 5 win more than 2bb/100 and in my experience that's not easy to do. The people that are profitable shortstackers would likely win a lot more playing with a normal stack.
I don't think there are many long term profitable shortstackers following rules. Again, I think the majority of the profits come from a few strategic plays that would be impossible for a spreadsheet, or a bot, to emulate.
2-4 bb/100 is great even for a human since you can do huge volume that way. The spreadsheet method can easily be applied to 8 tables at once, maybe more. 2 bb/100 (thats $4 as calculate by PT right?) could be $30/hr. 4bb is obv $60, which is more in line with what people I knew were making.
Also when short stacking, opponents at higher level games don't necessarily respond any more appropriately. In fact a lot of times they play worse against it because they have more idea what to do vs. normal opponents. Low level players like to call big bets too much, which makes them accidentally better vs. a short stacker. My friends claimed they were getting higher bb/100 at the $3/$6 and $5/$10 games than the $1/$2 because of that.
Not PTBB/100, just BB/100. PokerTracker couldn't handle all the data.
I suppose you could possibly control for play style, thinks like what Poker Tracker tracks. VPIP, etc., and of course buy-ins if you do some technomagic to figure them out. (I'm assuming someone who short stacks all the time plays differently than a normal player who just lost half their stack.) Not sure how good your findings would be though.
I think the one result most applicable in this scenario is the section with jam or fold tables based on hand and stacksize. Basically, they show that there are completely defined hand ranges when you should jam or fold when stacked at a certain number of BB at no-limit.
I've pondered doing a neural network, but for limit rather than no-limit. I still may, mainly because I think working on a bunch of inputs that are hard to quantify on the fly when playing would be fun to experiment with via the neural net.
Thanks for your posts on poker bots - although I have to admit I've basically quit playing online over the last year because of the proliferation of HUDs, data mining automation, and bots.
Oh, and for anybody interested in game theory and/or poker, you simply have to grab the book. It's a bunch of fun and not scary on the math for those on this forum.
The push/fold tables are another story, and I'll write about that soon.
It allows for bots in Perl and has a DLL interface. I've been meaning to make a V8 based DLL to write bots in Javascript for a while, but it's not a high priority for me.
P.S. Did anyone notice this guy has a Tetrinet bot?! That game made high school tolerable.
If you want to criticize neural nets as tricky and confusing to newbies with a steep learning curve, that is fair. But it's irresponsible criticism like yours that has given neural nets an undeserved bad reputation in the machine learning community.
there is no way to find out the relative significance of the input variables and how they affect the output response
Calculate partial derivatives of inputs w.r.t. outputs.
there isn't a way to cluster similar cases together to analyze what rules bind different decisions together
PCA analysis of activations of the hidden layer. Could also do k-means clustering of activations of hidden layer.
The random forest classifier looks interesting, I'll have to investigate that. Any suggestions for papers/tutorials?
"Calculate partial derivatives of inputs w.r.t. outputs."
Yes. There are other ways to interpret the output. See, for example, "Visualizing Higher Layer Features of a Deep Network." by Erhan et al 2009: http://www.iro.umontreal.ca/~lisa/publications/?page=publica...
It is true, though, that more powerful models are less explainable. But in return you have more compact modeling and training via gradient descent. This is faar faster than the combinatorial optimization involved in ensembles of trees. (Trust me, I've implemented both.)
there isn't a way to cluster similar cases together to analyze what rules bind different decisions together
"PCA analysis of activations of the hidden layer. Could also do k-means clustering of activations of hidden layer."
I concur. However, for 2-d or 3-d visualization, you should use the more recently developed t-SNE algorithm instead of PCA or other alternatives. t-SNE does waay better. Software (in Matlab and Python) is available at: http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE.html
Also, you should look at the JMLR paper (http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE_files/vanderm...) and supplemental material (http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE_files/Supplem...), to see the visualizations produced by t-SNE and competing methods. This qualitative evaluation by looking at pictures speaks for itself.
Also, one cannot find the partial dependence of the output on a given input variable.
If your model assumes that the output is a non-linear combination of inputs, then yes, it is hard to express the output in terms of a linear decomposition. But that was your choice of modelling assumption, presumably because linear models are insufficiently powerful to fit the underlying variations.
"The random forest classifier looks interesting, I'll have to investigate that. Any suggestions for papers/tutorials?"
Random forests were developed by Leo Breiman (RIP).
"Ensemble methods in machine learning" by Diettrich (2000) compares different tree ensemble methods. He concludes that boosting an ensemble of decision trees is better, except when the data are very noisy, in which randomized trees are better. (Boosting is when you focus on the examples that the model is currently doing the worst. Randomized instead works on random subsets of examples.) The main reason boosting is worse than randomized trees in the noisy case is because the AdaBoost exponential loss is sensitive to outliers. Which is to say, AdaBoost boosts the wrong loss function. Boosting an appropriate choice of loss function (perhaps a regularized log-loss) is probably superior to randomized trees in most circumstances.
"Improved boosting algorithms using confidence-rated predictions" by Schapire and Singer (2000) is a great introduction to boosting.
Around the same time Llew Mason and Jerome Friedman independently demonstrated that boosting is essentially fitting an additive model using gradient-based methods to select the features that have the steepest loss gradient. So you should follow up by looking at their work.
What about attacking 7-card stud, stud8, omaha, and o8? These games have more information to analyze than hold'em because more cards are visible. It seems to me that would provide more of an edge to a computer, which could calculate drawing odds faster and more accurately than a person.