AI Beats Four Top Poker Players | Hacker News Reader

AI Beats Four Top Poker Players

bbc.co.uk

243 pointsjeremyleach220 comments

Anyone that's interested in reading a more detailed account of the experiment can do so here:

http://www.pokerlistings.com/libratus-poker-ai-smokes-humans...

The above article spells out some of the details of the competition. The winrate (14.72bb/100) that the AI achieved over the 120k hand sample is almost certainly not due to luck. It is a huge winrate that most pros have to employ strong game selection techniques to achieve (only play against bad players).

Here's a layman's explanation of how a poker AI can be trained: http://www.pokersnowie.com/about/technology-training.html

And some details about the weaknesses resulting from how they've abstracted the game: http://www.pokersnowie.com/about/weaknesses.html

If it was paying rake like online players do, then that winrate would drop into single digits.

That's pretty impressive, the win rate and amount of hands played. It's certainly not a fluke. At the same time, these 4 pros, I've never heard of them. This AI and organizers should focus on HU specialists. Perhaps even invite players like Negreanu, Ivey, and more recent online pros who've made a fortune. Just don't let Hellmuth to play this machine, it would be a disaster.

These pros are considered to be top 20 HU online specialists, they would destroy Negreanu and Ivey over the long run at this particular variant (which they have played millions of hands at).

Back when I played poker (up to ~2010), Negreanu was mostly a celebrity, as well as a pretty good blog writer. He was no match for the best pro players, but perhaps he has dramatically improved.

Ivey was pretty good at managing his image, too.

Say that all you want, those guys play(ed? I'm out of the loop) in the big game, and...whoever you're thinking of, basically--didn't.

Conversely, not many of the celebrity players have had much success playing online. Ivey did have some good years but even he has been struggling for the past few years.

Almost all pros who have had success both online and live agree that live games a ridiculously soft compared to online games. The reason more online pros don't play live is because you have to live in Las Vegas or Macau to play a the highest stakes, or you have to be invited. Also, the big live games are usually mixed games that don't afford a huge edge to online pros who specialise in a few games.

I don't dispute any of that, I simply dispute whether or not Negreanu is a "top pro," or whether he would be a match for "the top pros."

He is a top pro. Not in HU cash. He couldn't hang with these guys at HU cash, at least not with his current skill in the discipline. But hey, poker is not that narrow of a term. It includes all kinds of disciplines, live and online, horse and stud and hold 'em, the list goes on.

Those same "top pros" who Negreanu wouldn't play HU Cash wouldn't sit in the big game with him.

Arguably bankroll management is the most important skill of a top poker professional, and that's maybe what big-game players are best at.

Also, a small caveat to all of your comment is that live games are soft compared to online games of the same limit. There is no online equivalent of the big game, or at least there wasn't when black friday hit and knocked me out of the professional poker scene.

There is no question that Daniel Negreanu is a very good tournament player, possibly one of the best. I don't think there is any evidence that he is anywhere near the top when it comes to cash games of any popular variety.

And while bankroll management is certainly an important and required skill to endure as a professional poker player at any level, it is by no means what differentiates high-stakes pros from pros at the lower levels. There are lots of small stakes and mid stakes players who practice sensible bankroll management, but who will never acquire the skill necessary to make it at the highest levels. If you want to get a sense of how much skill goes into playing poker at the highest level, you should watch some of Phil Galfond's strategy videos on YouTube (see for instance [1]). Poker strategy has come a long way since Super System[2] and The Theory of Poker[3]. Even players who were considered very good just five years ago, can no longer compete at the highest levels.

[1] https://www.youtube.com/watch?v=khdoSFCQ9iA [2] https://en.wikipedia.org/wiki/Super/System [3] http://www.twoplustwo.com/books/poker/theory-of-poker/

The "Big Game" which was run by Doyle Brunson, Chip Reese, etc had nothing to do with HUNL. It's mostly run hoping that some inexperienced player would drop in. It also almost always ran as a rotation of games to both bring in players who though they had an edge in "their" game and then exploit them in the other games.

The Big Game and online HUNL are both technically "poker" but they are truly completely different games.

I'm fully aware--I was a poker pro for a while in a niche (HUSNGs).

The post I responded to slighted Negreanu, saying he wasn't a "top pro." He certainly is. Not a HU cash pro, of course, but a top pro? Absolutely. A match for the "best pro players?" Certainly.

The "best pro players" in a given niche wouldn't have sat in the big game, and he wouldn't have sat in their specialty. Overall he's the better poker player.

Parent's comment about the big game is accurate; they were all friends waiting for some whale to show up, so playing there doesn't mean you're one of the best players in the world, skill-wise.

Up to 2010, Negreanu had some up and down and as poker theory developed, his game showed a number of pretty big flaws--which he himself admitted. Sure, he was a competent player, so calling him just a 'celebrity' might be misleading, but there were many players in the top online scene who did a lot more of volume and analysis that Negreanu, who pushed more edge and who leaked much less.

I certainly agree with you that Negreanu is a top pro. I thought I was responding to the parent post from marklgr. Current threading looks like I missed a couple of levels.

Andrew Ng posted a comment about this on facebook: "I'm thrilled about Libratus' Poker triumph--this is a huge step for AI. Othello/Checkers/Chess/Go were theoretically solvable with minimax tree search and sheer computation; but poker, which requires bluffing, needs sophisticated modeling of your opponents and new algorithmic principles. CMU's Tuomas Sandholm has also (in a private email) promised to publish their algorithms, which I look forward to. Congrats CMU!!!" https://www.facebook.com/andrew.ng.96/posts/1260889373966967

I'm surprised that Andrew Ng made this claim. The strategy that was built for Libratus' predecessor did not do sophisticated modeling of the opponents, or use new algorithmic principles.

Poker is solved using a very large game tree, just as with the other games. The structure of the tree is modified to support the notion of hidden state, but beyond that it is essentially the same as the other games. The structure for representing hidden nodes was developed in the 1950s by Von Neumann. Most of the algorithmic innovations related to how to update the game tree.

My guess is that the primary innovation for the Libratus strategy was that of scale.

This was my thought too. Poker is quite "solvable" meaning, whenever you're confronted with a decisions -- there is always a "correct" answer which does not have to depend on the other players' behavior or style. And you can find that answer by simulation, or game trees, and other methods.

It's also important to keep in mind that the best AI can still lose, and the worst AI can still win (and everything in between). Poker involves randomness, obviously whereas chess/go/etc does not.

That depends on what you mean by "correct". Sure, you could theoretically find Nash equilibrium of poker and by playing the equilibrium strategy, you can ensure you won't lose. But that does not mean this is the best strategy to use at a given table against the given opponents, who (being imperfect humans) almost certainly do not play the equilibrium strategy themselves. And, by playing a proper nonequilibrium strategy, suited to the specific players, you can win more.

The usual way these games are solved is to create an "abstract" game which is tractable, find the Nash equilibrium, and map state in the real game back to the "abstract" game. In the limit, the solutions for a well designed "abstract" game will converge to that of the real game.

You are explaining how current algorithms try to find the (approximate) Nash equilibrium (and those algorithms are far from perfect; as noted in the recent DeepStack paper, current abstraction-based programs are beatable by over 3000 mbb/g, which is four times as large as simply folding each game). But my point is that even the (exact) equilibrium strategy would not necessarily be the best strategy against given non-equilibrium-playing players.

Yes, you are correct on every point. Opponent modeling and exploitation is significantly more difficult than coming up with a Nash equilibrium to an abstract game.

>there is always a "correct" answer

That's wrong. Even when you're holding a good hand, your opponent could hold a better one and reading them is a key element of poker. The opponent's hand is an important variable to decide whether you hold the winning hand or not.

If you look at the experiment in detail, you'll find that it was set up in the AI's favor.

>When a hand was all-in before the river no more cards were dealt and each player received his equity in chips.

While all that is less important when you can avoid all-in situations, the main statement -that the other player's behavior is irrelevant- is still wrong.

>>If you look at the experiment in detail, you'll find that it was set up in the AI's favor.

Could you elaborate on this ?

Links were posted here: https://news.ycombinator.com/item?id=13535714

As expected, the AI is good at making technically correct decisions and "draining money" from a table by playing hands with sufficient data almost perfectly.

However, in decisive all-in situations with little information available, it supposedly wouldn't do so well, regardless of all the learning, but that's what it often comes down to.

>Nash Equilibrium is a strategy which ensures that the player who is using it will, at the very least, not fare worse than a player using any other strategy.

How do you make this work for situations that can cost you the game in one hand, with little information available? Without observing the opponent's behavior you can't, and for the AI that means it can be forced into making bad calls by playing aggressively, unless the game mode allows for avoiding such decisions, which was the case in this test.

The no-limit hold'em game tree has over 10^160 states.

Even limit poker is too large to solve directly. In 2015, limit poker was essentially solved with a new technique in game theory that allowed them to find a simplified model.

http://spectrum.ieee.org/automaton/robotics/artificial-intel...

Chess has a lot of possible states too. (Yeah, I know that's different.)

Heuristics make analysis practical, and well chosen ones make the difference. This isn't to minimize the accomplishment, but rather to say it has strong similarities to other AI games.

You have to model opponent's behavior though. For example, the probability opponent will fold when AI bets a certain amount.

This is sort of correct. In a Nash equilibrium, the other player is modeled as also playing a Nash equilibrium strategy. Nothing more.

> Poker is solved using a very large game tree

You mean Libratus' strategy used a very large game tree. That is not the only strategy. Take a look at research from the University of Alberta [0]. Also, I'm not certain Libratus' strategy can be simplified to "very large game tree" as I haven't seen the paper, yet.

While finding a Nash equilibrium means no other player can beat you, it doesn't mean you're going to make the most money in a big ring game. A different strategy might lose money to an equilibrium player, but exploit a different, weak player so much that it's worth the loss.

[0] http://poker.cs.ualberta.ca/publications.html

Key example of that Nash point is Rock Paper Scissors. You can't exploit anybody playing 100% random, yet that's the Nash Equilibrium. RPSAI competitions tend to have 100% random players perform quite poorly

Do you mean to say that 100% random RPSAIs have a lower winrate vs humans than RPSAIs that learn and exploit human patterns? Surely a 100% random RPSAI doesn't have a poor win rate against any other RPSAI?

A 100% random RPS AI doesn't have a poor 1-on-1 win rate against any other RPS AI, but it absolutely can have a poor rate of winning tournaments, if "poor" is defined broadly enough. For a tournament that pays cash to the top 10%, most human players would consider anything in the bottom 90% to be poor, which would include a 50% win rate from a random AI.

This happens because some entrants aren't 100% random, and the worse of them can be exploited by the better of them. What happens is that the results involving any random AI essentially degenerate into noise, while the tournament is really contested between the nonrandom entrants and will be won by the one of them with the best strategy.

Put another way: to win or place highly in a tournament, you don't just want expected win-rate, you want variance. If there is no difference in reward between a 50% win-rate versus a 10% win-rate (both are far out of the money), but there is a big difference between a 50% win-rate and a 90% win-rate (the latter wins the tournament), you will seek the 90% at the cost of potentially ending up at 10%.

To add something, and to clarify (though I don't think you've made any mistake here, but many people misunderstand this subject):

The 33%-each Nash Equilibrium is the mixed strategy Nash Equilibrium of the micro game (i.e. a single round of RPS, averaged over all possible randomizations).

This is in no way the Nash strategy of the tournament game, which is "win the tournament given a pool of unknown participants and a set of rules for whom you face when". You have to add additional assumptions (e.g. that everyone else is going to play the uniform random strategy) in order for uniform random to be the Nash strategy for the tournament game.

If the pool includes players who deviate from the single-round nash equilibrium strategy, there is opportunity to exploit them (and in doing so, open yourself to possible exploitation). This is why pure random play can often perform very poorly at the tournament game.

> You have to add additional assumptions (e.g. that everyone else is going to play the uniform random strategy) in order for uniform random to be the Nash strategy for the tournament game.

Isn't that literally what a Nash Equilibrium is though? It's my understanding that if there is players playing exploitably in the game then it cannot (by definition) be a Nash Equilibrium, so the Nash strategy may no longer be the optimal or maximally exploitative one.

Correct. Nash equilibrium exists only if everyone takes the equilibrium-seeking strategy.

Similarly, poker tournaments are only profitable for a professional because so many weak players will register -- the "dead money" so to speak.

Thanks, I understand now what you're saying.

> RPSAI competitions tend to have 100% random players perform quite poorly

Not as much "quite poorly" but more specifically, they will provably land at exactly the median position in the ranking (let's assume there's only one pure random bot in the tournament, no reason to have more than one, but the argument also works with multiple).

While it's impossible to win more than 50% of the time from a pure random RPS bot, it's also impossible to lose more than 50% of the time.

So all the other AIs will on average score exactly 50% against the random RPS bot. Whether they end up in the final ranking above or below this median line depends on how well they do against each other.

A Nash equilibrium (or very close approximation thereof) does sound like it would reliably accumulate a chip lead over a very large number of hands against four top level human players though. They'll tend not to have readily exploitable weaknesses and won't be equilibrium players either.

Fair enough. I'd certainly take a conservative, equilibrium approach when playing against a very skilled opponent.

it won't be long before we hearing more headlines like:

"AI beats top 10 hedge fund managers"

"AI run hedge fund blows up due to black swan event"

regardless it's an incredible feat. It really casts questions into what our edge as humans are which is slowly disappearing and we didn't even need to put a brain in a jar and hook it up to a computer....it's deep learning reinforced algorithms that is appearing to outlearn, outthink the best of humans.

I just can't emphasize what a monumental period in history we are at. Humans are producing specialized algorithms that learn and hold information about the deep web of relationships between myriads of parameters to produce superior performance than humans.

It's almost like we've uncovered ways to automate our intelligence very much like we've been automating human and animal labor in the past couple centuries.

So the question is, how does an average joe hacker like me exploit and leverage this wonderful thing called deep learning? I'm not interested in reading PHD papers with advanced calculus.

I want to have a map of what AI, ML, DL, NN methodologies to use and when and who to hire based on that. This is no time to be a luddite and don't count on basic income from appeasing the masses anytime soon. Much like people took the most hit in the early rise of industrial revolution, our generation and immediate generation will be hit the hardest.

> "AI run hedge fund blows up due to black swan event"

Knight Capital, 2012:

https://www.bloomberg.com/news/articles/2012-08-02/knight-sh...

How many of these hedge funds running on AI are out there in America?

I'd think one would make a lot of money shorting such outfits.

That would only work if you'd expect these events to be frequent enough that you wouldn't have to short for very long or else the costs of your short would outpace any possible profit very quickly.

Only if you can time when they're going to fail. It's not enough to be right, you have to be right at the right time.

I know with high confidence that these AI run hedge fund outfits reverts to a zero sum mean. Doesn't mean that there won't be one or two that beats the market for sometime as a whataboutism. The timing is a major issue, it's tough to at what year these AI hedge funds fail.

The other capital that blew up from the comments suggest it was more of a human error that led to deployment failures which is exactly why AI hedge funds is such a paradox. At the end of the day, it's humans that's deploying and building the model to be approximately right with high degree of accuracy but in the day trading environment it is a poor model for success-not only do you have to be right your monetary exposure must be right...."but proper money management and you will be fine" said everyone who took part in this zero sum game.

I don't know, prove me wrong, I'm sure those ivy league engineers on wall street aren't getting paid dimes for what they do.

I believe the poster meant a machine-learning system blowing up (catastrophic trading loss) due to a black swan event that the system wasn't trained to deal with.

Knight's blow-up was a process/sysops failure, not due to their strategies going wonky.

[As a side note I believe their core strategies were human-analyst designed, not ML based, but I could be wrong.]

Wouldn't process & sysops failures count as blackswan events if the AI or ML algorithms weren't trained to deal with those types of hiccups?

Not quite. This was merely a simple deployment mishap due to repurposing a flag which triggered deprecated code paths that should have been gutted out.

That wasn't really an AI failure:

https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...

Even if a human deployed it, considering that everything automated by a computer can now be called AR by PR agencies it is an AI failure ;)

> "AI run hedge fund blows up due to black swan event"

Why's this an argument against AI run hedge funds? Funds run by humans also blow up due to black swan events.

> So the question is, how does an average joe hacker like me exploit and leverage this wonderful thing called deep learning? I'm not interested in reading PHD papers with advanced calculus.

Well reading paper is a must to get to deep learning. Those papers may not be that math heavy once you are used to it. Most of the time, it is about network architecture and loss objectives.

>AI beats top 10 hedge fund managers"

Hedge funds are already using A.I. successfully. I talked one manager who's team was trying to layer successful individual neural networks into a "bigger brain".

Been there, done that (well kind, of, via several small specialized nets). Actually a pretty good trading strategy, but works better if you actually have capital to use it....

I was under the impression that you need some capital, but not so much that you move the market with your trades. True?

eh, if you've got the money to be worrying about moving the market, you're using brokers' advanced execution algos and darkpools.

> if you've got the money to be worrying about moving the market, you're using advanced execution strategies and darkpools

These aren't silver bullets. Both cost more, in fees, and have a habit of leaking order information across informal channels. They're also, generally, slower--there are, on average, fewer buyers and sellers for a given security in a dark pool than in the open market.

IDK that much about the finer parts of trading, since I just approached it from a ML perspective.

What I do know is that it wasn't worth it for me personally to occupy my GPUs. By contrast, larger trading firms would probably like it.

probably not going to get any details out of you but worth trying: can you give any pointers to implement such strategies and the minimum capital involved?

A Google search result for "machine learning stock trading":

https://www.udacity.com/course/machine-learning-for-trading-...

well to be honest I never bothered googling that exact term but I was more interested in the details around his deployment of smaller and successful neural networks as part of a portfolio.

Basically, certain neural networks perform better at predicting different factors at different market conditions. You can string them together to get a decent trader, enough to make a profit, but not enough to justify hogging your GPUs.

As in the profit generated isn't enough to sustain the electricity cost of running the GPUs? Are you able to scale up with more initial investment? How much did you invest and what kind of return on investment were you getting?

Not enough to justify using up the compute capabilities of my GPUs since I need them for research.

I averaged a return of 7-13% a year. With less than 1000 in capital, it wasn't very profitable.

I don't see why it shouldn't scale.

Rule #1: It's always better to trade with OPM (Other People's Money)

> "So the question is, how does an average joe hacker like me exploit and leverage this wonderful thing called deep learning? I'm not interested in reading PHD papers with advanced calculus."

FFS, if you're not willing to read a paper with BASIC CALCULUS (it's HS/college, not advanced like fractional), then I'm not sure machine learning is the right place for you.

Sure it is. Much like most developers don't spend their time reading on computing science papers to do their job, I question why such academic rigour is to be demanded from somebody trying to capitalize on the arbitrage opportunity.

Math is good but my time won't be best used if I have to learn calculus all over again just to begin understanding the linguo. Rather have a generic model of what to use and when, hire those that have that capacity to dive deep when needed and implement the expected business outcome.

Because this is the language of the domain. Much like code is the language of programming. This isn't academic rigor or lingo. This is how things work.

To me, your comment reads similarly to "Much like most business people don't spend their time reading code to do the job, I question why such academic rigor is to be demanded by someone who is translating business requirements into database operations".

I think we are after different things. My goal is not to understand the underlying technology at the academic level but just enough to make money by utilizing the people that do, just like any good capitalist.

You don't need to be an oil rig worker to understand where the opportunity lies is what I'm saying. It's possible to understand systems from a blackbox point of view and still be able to exploit them. Afterall, this is what we do whenever we hire a new developer, it's impossible to know what they know so I leave that up to smarter and talented people while I allocate the capitals and set a vision for what's required to capitalize on the business opportunity that AI presents with it's superior performance, scalability and economies to scale.

You outsource what you don't know or can't do and if you can't afford it you do end up doing it yourself. This is why capitalism works. It's not necessary for you to dwindle with all the underlying complexities when you model the business opportunity in terms of basic expenditure and investment returns.

If you want to be a venture capitalist, then you need to bring money to the table....

I don't know why you expect to get ahead by hiring others to do your thinking for you. If you don't have a basic understanding of the technology you're hiring people to implement, they'll just pull the wool over your eyes, sell you something that doesn't work and make away with your money.

Then hire someone who understands it.

> "...academic rigor..." "...calculus.."

Dude, I'm not even asking you for anything that difficult here, how would you like it if I walked in and said "well, I don't need to learn any programing, my time is better spent magically making it work".

I found calculus and academia generally very difficult. I'm happy to know that you don't suffer from this terrible affliction which makes studying difficult.

You don't need to know programming if you understand your workers and they understand it enough to convey to you the time & resource constraints.

I have a million dollars for this project. What are the features and functions that will maximize my returns? Hire a couple of smart engineers who already holds the technical knowledge, able to communicate when asked all the pitfalls, potential roadblocks and unknowns so that their productivity is maximized. Set expectations on how they will be able to trade their time for money. At the end of the day, it's a naked call option with limited risk and unlimited reward by utilizing the correct knowledge workers.

Why spend time learning the reference when you can look it up or hire somebody who already has a distilled version in their head which you can utilize?

I think people are just saying that if you don't undertstand the technology a little bit yourself you'll never know who to hire. Maybe somebody looks like they know what they're talking about, but they don't. Maybe somebody really knows what they're talking about but they can't put that across very well. Maybe someome is simply unscrupulous and wants to milk you dry and will dazzle you with charts and graphs that they know you don't quite understand.

It's, like, inadvisable, to leave yourself vulnerable like that.

Absolutely. I know just enough software engineering to bullshit my way in front of a nontechnical person. If any such person hires me to do serious dev work they'll be in a sea of trouble.

Okay, here's a million dollars a year budget, who do you hire ? If you've read papers and know about deep reinforcement learning , you'll know who to hire.

I hire the guy that read the papers and know who to hire :)

In an orchestra, the conductor is not replaceable but the instrument players are. Similarly, someone who allocates his or other's capital cannot be easily replaced very much like the worker's.

Your engineer leaves there are 100 others ready to take the place. Business owner leaves you'd have to find a buyer to keep everyone going.

This is different from a CEO funded by VC's where you are still replaceable but nevertheless a lot harder to replace than the cost intensive worker.

Being on the profit generating part of the business and cost intensive part of the business helped me understand why such discrepancy exists-those who are able to read and perform in chaotic and uncertain environments will always be valued than those who directly affect the margin's as a result of the monetary figure attached to their time which is always far more costlier because cost centres do not generate new revenues. Your top sales guy makes $9 for every $1 spent on him (+$8) where as your top engineer costs $3 for every $0 he generates (-$3).

tl;dr: revenue generators are king while cost generators are easily swappable due to the high supply of it as a result of it being a far safer and financially stable perception.

Hint hint: you're the most replaceable one in this setup

Funny, I know more people who are asking about machine learning than people who are experts. So who is in high supply?

Machine learning experts are in higher supply, your anecdotal experience offers little weight. High demand creates a high supply which drives wages to zero.

For the foreseeable future, machine learning experts will be in sufficient supply for them (i) to have wages which are considerably above zero and (ii) have the ability to raise capital from people that don't think they're commodity labour

(it cuts the other way too: if an oversupply of labour in a field as difficult-to-learn as ML does arise, the demand shortage causing wages to fall is almost certainly because ML techniques aren't giving companies and traders as much of a financial edge as they hoped for...)

Added bonus for the ML experts: in a lot of the possible areas they can choose to work in, their individual marginal contribution to the company's profitability is at least as quantifiable as that of a salesperson. In many areas of finance you can quantify the impact of an individual line of code!

It's not enough to hire a machine learning expert if you are planning on building out your own capable ML operation confident in competing against the best and the brightest at Azure, Google & AWS. You'd need a team of ML experts.

On the other hand, it's easier to piggy back off the hard work of highly paid experts in this space now commoditized into 1/10th of a cent per application of an ML algorithm from Microsoft or Google or IBM.

Commoditization is going to be in full force as businesses realize they can just have a junior developer to plugin IBM Watson or Azure than hire an expensive PhD student who may know everything under the sun but cannot compete against an army of highly paid experts completely under the control of a highly collusive labor market.

I think that idea is largely responsible for the decline of HP, IBM, etc.

Karpathy was offered in excess of a million out of school, I'm going to assume that you don't know what you're talking about.

Silly google, they must've skipped Econ 101 since they hire tons of engineers and are doing TERRIBLY!

And how many Karpathy is there working for non-Google? Mariah Carrey made $140 million dollar record deal with Sony Music, there's boat load of money to make in the music scene!

So your SOLE contribution is capital, right?

Why would I go to you and not a VC firm? I sure as hell don't want to work for someone like you....

(oh and there are plenty of others like Karpathy, like Facebook, Baidu, Microsoft, Uber, Amazon, a metric fuckton of startups, etc. who are being paid ridiculous amounts of money to build nets)

Ha! That's an amusing interpretation of how supply and demand interact to find an equilibrium price/quantity point.

A typical Econ 101 textbook says:

    higher demand --> higher price
    higher price --> higher supply
    higher supply --> lower price
    lower price --> higher demand

In Econ 101 we pretend that cycle eventually reaches an equilibrium. In grad school we analyze the dynamics.

But even if we believed your demand causes supply causes zero price model, why is that unique to engineering and not business acumen? And why do programmers get paid anything, don't they work for free?

Yes and if you've actually paid attention you will see that a high salary signals other workers to flood the market with a lower salary thus starting the race to the bottom :)

Programmers have largely driven themselves to zero, just take a look at the workers on freelance websites and how much difficulty a native English speaking freelancer has against an army of commoditized labor.

There are still six digit earning engineers and always will be but you can't look at that as a metric for the massive commoditization that has occurred with generic programming in the past 15 years.

There's some premium for face-to-face, but it turns out that even with remote work, price is still a good indication of quality.

I'm a freelancer and fully booked, so maybe those low cost competitors aren't so competitive after all.

Another nonsense implication of your theory: it's better to have a low wage, so you stay under the radar and don't signal other workers to enter the market.

I think you are just trolling at this point.

https://image.slidesharecdn.com/requirementsengineering-1402...

Guess who's getting replaced once the VC figures out who's doing the work...

> Your top sales guy makes $9 for every $1 spent on him (+$8) where as your top engineer costs $3 for every $0 he generates (-$3)

In this case you should immediately fire all your engineers, your CFO, and your COO. Because why would you want to employ a team of engineers who are costing you money but bringing you no value?

You should also fire your cleaning staff, who have never closed a deal in their lives, plus the plumber who most days dosen't even come in.

I'm glad you posted it because it shows how little people understand their engineer's financial importance.

Granted, engineers are required to create products and maintain it. You need product to sell.

But at the marginal level for every dollar your sales person makes your engineer is not. You might argue but the product is generating revenues but that's not what drives a sale. A sale is a function of value derived from the product and the price paid for it. Engineers aren't driving the sale unless the product itself is a developer tool. But even in that case the top level business controllers will always have the final say. In the end, buying such developer centric tool is about minimizing cost expenditure.

There is critical proprietary knowledge and experience that is formed from a sales person that is driving revenues. Businesses do not want to lose the hen that lays the golden egg as they are hard and expensive to replace.

Engineers on the other hand are easily replaceable and are expensive because they cannot generate revenues. They are not efficient when pulled away to a sales meeting from the work they are doing. Their knowledge and skill costs businesses time and money with no way to get it back directly. It must be sold, money collected and redistributed to all the payrolls. That's the main risk a business owner or a CEO is dealing with where as the engineer collects a cheque at the end of the month with little to no concern or exposure to that risk.

Society rewards risk takers disproportionately at the corporate level. Even if it's tough to measure to the Board and investors losing a C-level executive is always going to be more impactful than a senior engineer who has far more workers to replace him as a result of being cost intensive.

Find me an engineer that can code and sell, now that is a truly rare hybrid, a mewtwo, but it will still be more expensive than a guy who just sells (and does well).

A poor salesman understands this better than anyone, his weight is worth the revenues he generates. For engineers it's the efficiency / dollar or output / dollar that they are competing against which is always headed towards commoditization and any business would replace them with an AI that can code if they could if it cost less.

If good engineers are dime a dozen then why do they command >200k salary? I don't think you understand how much skill and practice goes into making a good programmer. Your ignorance is truly astonishing.

I find this thread mildly amusing. It's like both engineers and businessmen thought themselves irreplaceable. And yet, 'The graveyards are full of indispensable men.'

And for every engineer earning 200k salary in SV how many are earning less than that? I don't think you are seeing the bigger picture of the labor market instead basing it off of biased observation.

>Hire a couple of smart engineers who already holds the technical knowledge

Then exactly what value are you bringing to the table? Money? Sales skills? If that's the case then why focus on ML/AI technologies? Any company needs money or sales expertise.

Pretty sure it's going to be his unique talent of making employees feel valued within the business, his charisma and sharp ability to clearly convey his business ideas and cohesive argumentation skills when asked about the giant holes in them ...

How do you suggest you take advantage of an arbitrage opportunity if you have 0 informational insight?

Because the arbitrage opportunity isn't present in the academic papers but at a higher level involving business problems.

I say this is an arbitrage opportunity because AI is essentially commoditizing human intelligence. It didn't take much brains for people to realize cars could do away with horses. You don't need to understand how a car works down to the formula. It's enough to view it as a black box and still know all the outputs and constraints associated with it in order to orchestrate capital that allows you to seize the business opportunity.

Again, the arbitrage is buying something for cheap and immediately selling it at a higher price...if you can't understand why a deep level understanding of the academia is not necessary in order to capture the opportunity then less competition for me which is great.

I think just read the Wikipedia articles about it. Also, the Microsoft documentation on Azure Machine Learning and the Amazon one on AML are good high level overview.

That should be enough for you to have an idea what expertise you'd want to hire for.

Granted, it sounds you're just looking for ML scientists or ML engineers. Data Scientist might also be able to help you out, so that's what you want to hire.

And yet the people who made fortunes in the auto industry were all car makers? What an odd coincidence.

But let's indulge your point of view for the sake of the argument. So you've got a black box AI that does something. You know what inputs it needs and what outputs it produces. How do you arbitrage it? What creates the price discrepancy that allows you to arbitrage at all, given that your incapacity to understand the black box renders you unable to gauge its value beyond what the market is telling you?

In that case, why not trade any random commodity?

The arbitrage is on the labor. Find someone who can build x for y and then sell it for z. The arbitrage opportunity comes from creating or obtaining a market you control. That market can be everything from a platform(Apple) to a collection of sales channels(Oracle) to a monopoly IP position(Microsoft). You do not need find people because they come to you because you control the market.

> In that case, why not trade any random commodity?

Its much easier to gain control of markets that are new as opposed to established ones.

You nailed it. AI advancement leads to arbitrage opportunities in white collar labor as did mechanical automation lead to arbitrage opportunities in blue collar labor which leads to group of giant firms that collude to keep wages low as a result of the Nash Equilibrium. Apple won't pay workers more than Google does and vice versa. Starting a tit for tat would drive wages high which would increase uncertainty and risk for market monopolies.

The answer is that many ML solutions are leaky abstractions. See: https://medium.com/@karpathy/yes-you-should-understand-backp...

Remember "idea-men"?

"Listen I have this great idea for a web app, it's going to be like the Facebook for hardware stores [or whatever] and people can rate/vote/tag ... I got the whole idea worked out, I just need someone to program it for me" etc. Every programmer has heard these kinds of proposals many times.

One (of many) problems with this kind of proposal is that the idea-man, unable to program (and not wanting to learn) has no idea about the complexity of what they're asking. And even if they think they have the whole idea worked out, not being able to program means they probably missed a lot of shortcuts, possibilities and best practices. The worst ones hand-wave this with the possibility of a fresh outsider-look on things! Except that the whole (fully worked-out) idea probably needs to be reworked entirely before it's even feasible or competitive. The programmer would probably have been better off without the idea, cause that's where it started to go wrong.

Similarly, if you believe math isn't worth your time but instead want to hire ML experts to implement your ML-related ideas (formulated in terms of "expected business outcome"), puts you in a very similar role.

(Tongue-in-cheek) I got a really great idea, the expected business outcome is: make lots of profit. It's fool-proof. I checked with some of my people and they agreed, making profit is a solid idea. I just need some business dude to implement for me. I'm not really sure what sort of business person, my time can be spent better than learning about subtleties between sorts of business people. I just need to hire a good one. But remember, the idea was mine first.

I'd argue that reading academic papers on DL is hardly the only or even a good way to get into DL for an average Joe Hacker. "... if you're not willing to read a paper with BASIC CALCULUS ... then I'm not sure machine learning is the right place for you" is the exclusion attitude. It does not have to be that way. I'm glad there are projects like http://www.fast.ai/ which aim to change that.

It's just that AI researchers are exploring and finding every single field where deep learning algorithms perform better than humans. Play poker, run hedge funds...

But for instance understanding and specially producing meaningful language, let it be natural language or programming language? I hope not so.

Because otherwise, your average joe hacker might as well shut down their IDE and say good bye.

Looking at the recent leap in translation between natural languages, it seems that producing meaningful language about various situations is not far off.

Well, but that is not really "production", because you need the language to translate from.

> It's almost like we've uncovered ways to automate our intelligence very much like we've been automating human and animal labor in the past couple centuries.

Actually, a lot of mental work isn't based on intelligence but rather training in pattern recognition, thinking faster, translation of information between different coding systems, and developing instincts about the behavior if complex systems. This isn't very different from manual labor when you think about it.

hmmm interesting way to put things. I often held such cynicisms about "intelligent" and "professionals". At the end of the day it's still labor, white collared, but none the less value is trapped in the blackbox which are humans.

Now it appears the essence of that is being distilled into parameters for AI algorithms which can produce superior output by teaching it with decades of professional human knowledge which maybe dumbed down with simple machine learning.

>"AI beats top 10 hedge fund managers"

Renaissance Technologies already is one of the top hedge funds in the world and it's fully automated.

[1]https://en.wikipedia.org/wiki/Renaissance_Technologies

> It's almost like we've uncovered ways to automate our intelligence

I wonder if humankind will be the creator of a new form of life. One that is not bound by chemical processes. One that moves at the speed of light and spans the whole planet. This being, or these beings, with access to billions of sensors, will know everything, see everything, hear everything. They won’t be limited to visible light, nor to the frequencies we can hear. There are no limitations except for what sensors can record. They will be able to interpret, predict, plan. They are not bound by time as they themselves don’t physically age. They will use vessels to carry out tasks and build the infrastructure they need to live. In comparison, we will look like single-celled organisms. So dumb. Until we disappear.

> So the question is, how does an average joe hacker like me exploit and leverage this wonderful thing called deep learning? I'm not interested in reading PHD papers with advanced calculus.

You've got a lot of comments like "if you're not willing to read a paper with BASIC CALCULUS ... then I'm not sure machine learning is the right place for you" to this, but I second you on this. I do have a PhD in CS and not afraid of calculus in papers. I just don't think reading PhD papers is the best path to getting practical results in DL/ML. "Not right place for you" is the sign of the "exclusiveness" problem.

I'd recommend checking the following course by Jeremy Howard and Rachel Thomas:

http://course.fast.ai/

It is designed to make DL accessible and achieves its goal very well. I'm not sure if the average Joe Hacker will be able to write top-pocker-players-beating program after this course, but it definitely gives enough insight and practice to get started in DL and be able to design and code solutions in a large number of problem classess.

I mean, by definition Black Swans can't be anticipated. So, no matter if it's an AI or human, black swans will throw you over.

But, an artificial intelligence probably has different black swans than humans, as their perception is inherently different. A tweet from Trump might not surprise us (anymore) but to an AI player, it might not be evident that a small amount of text from one Human can cause an uproar on the market. And it might not even have access to the relevant data (Twitter) at all.

I suspect Libratus' overbet frequency is overfit to this particular reduced-variance game format. In a normal game, the opponent doesn't take chips off the table after winning a hand and might stand up at any moment.

It's hard to know how much that affected the strategy, but in the Reddit thread, the human players said the overbet frequency was what they were most surprised by.

Maybe overbetting is better EV and humans just don't know it yet. It will be interesting to see if this changes the game at the higher stakes Heads Up tables

Yes, but expected value isn't everything. A good investor considers the ratio of expected return to variance of return. In a normal game, the variance on an overbet is enormous and therefore the denominator of its Sharpe ratio [0] is large and drives the value (or should I say "score" so as not to get confused with "expected value"?) of that action down.

You'd happily go all-in pre-flop AA vs KK. On the other hand if you got 4-bet pre-flop by a 22 and you're holding AK, you might ask, "Check it down?" This is assuming the opponent really likes small pocket pairs and will call an all-in, etc.

[0] https://en.wikipedia.org/wiki/Sharpe_ratio

> you might ask, "Check it down?"

In a heads up game? Really? "Check it down?" ???

The example applies more to a ring game, where you might be going after a particular fish and don't want to get involved with a better player. Still, I think it illustrates the principle of trying to keep the variance down.

Also, if you're first to act on the next round, it might be worth asking, even if you don't think they'll agree.

That's collusion and it's not cool. No top pro would ever seriously ask to 'check it down'

Kelly's criterion and the sharpe ratio is not relevant for the format played in these human-AI heads up games. They play with an unlimited bankroll.

I'm not talking about a tournament. And yes, it could indicate collusion. But most of the time, it's just a friendly low- or mid-stakes game and the players are tired of thinking.

My point exactly about the unlimited bankroll. The experimenters may not have realized that an unlimited bankroll would significantly affect the strategy.

Theoretically the best way to maximize an edge is to just wager larger amounts at all times. This has a beneficial side effect of deviating from 'standard' human play which can be further exploited, although I doubt this the intention of the AI.

Maximum expected value is not always optimal. Bets are an investment with uncertain returns. In normal circumstances, portfolio theory helps us optimize our investments. This reduced-variance format removed the budget constraint and made portfolio theory irrelevant. Overbets made sense to the bot who had only trained in this particular format, but not to the humans who have trained for years with a budget.

Reddit AMA from the live match:

https://www.reddit.com/r/IAmA/comments/5qi3i9/we_are_profess...

"Each night after the play ended, the Pittsburgh Supercomputing Centre added computations to sharpen the AI's strategy."

This sounds more like an "advanced chess" setup, where a human teams up with an AI to play. The title of the article should really be "amateur poker players + AI defeat professional poker players". The real test would be if the AI self-corrected over the length of the tournament, without human intervention.

That's only true if the humans made decisions about what computations to do after each day's play. The computations could have been determined in advance, eg. always run some algorithm to analyze the opponent's play for weaknesses. That's not cheating because the human players get to do the same thing. It's impossible to enforce a "no thinking outside the game room" rule for a multi-day tournament.

I dont think it's "cheating", but it's not just AI that's beating the humans. If operators are allowed to change/tune the AI's algorithm, the human opponent should get to replace himself with another player.

or conversely, operators shouldn't be allowed to tweak the AI logic, but if they program the AI to tweak itself, like review its moves and adjust its algorithm automatically, that would be reasonable, then the AI and nothing but the AI is responsible for the victory.

I don't see why you call it "operator tweaking" even if the computations were determined in advance. You're asking for the poker bot to be scripted to log into the Pittsburgh Supercomputing Centre on its own and run overnight jobs there. That idea sounds extremely brittle and the challenge of making that script reliable has nothing to do with poker.

"He added that the professionals had been sharing notes and tips in an effort to find weaknesses in the AI's game-play."

This doesn't sound like they had the goal of conducting a fully controlled experiment here, but it's still interesting none the less.

Especially given that the professionals were live streaming and kibitzing with chatters on Twitch during the event.

Once this is out, it's going to be have a big impact on online poker games.

They way things are going with AI. You have a good algorithm, you can get rich very quickly by being a one man business with hardware rented in AWS.

Libratus AI player is modelling it's human counterparts and predicting how they think to outsmart them.

When Google started, they got the page rank algorithm and distributed algorithms good enough to run on shitty unreliable cheap computers. They are well on the way to become the world's largest company overtaking Apple someday.

Their ad algorithms already know that I am applying for a house loan and are blasting me with ads every fucking page I visit on the Internet.

I can totally see Google and Facebook personalizing ads per person and taking advantage of the person's vulnerablaties. Like psychologically modelling them to make them click ads and buy random shit. I can see the start of ultimate God algorithms for marketing.

The ability for AI to create drug like experiences for us that we can't stop craving.

Not if you have uBlock (http://ublock.org/) installed.

Do a search for "long term" in online poker and you'll find that the suggestion for players to determine their level of play based on that "long term" is something like one million hands. That's running 4 tables for full days of poker over a long period of time. For sure it's playing more than 4 players.

The probability space of poker is such that 4 competitors isn't going to tell you much.

And defining "top" is difficult because "top" may be more celebrity than anything. Everyone has their different objectives. If you are "top" then you sure as didn't get there by building a case history against AI poker bots. Give these guys a chance to adjust, and give them a chance to figure out why the effort to adjust might be worth bothering with.

They played in pairs using mirrored hands and did equity splits when all in. This eliminated not all, but a large amount of variance.

From the article:

> A poker-playing AI has beaten four human players in a marathon match lasting 20 days.

20 days seems like plenty of time. No ?

No, the number of hands is relevant here, not the number of days played.

They won 14.7 big blinds per 100 hands, playing 120k hands. That's a pretty good win rate and a decent sample size but I don't know how to calculate the p value

Can someone clarify why exactly is this being presented as more impressive than beating Chess masters or even Go masters?

People talk a lot about number of states in poker, but the hand and visible cards can be easily (to a statitician) reduced to a scalar "probability of having the best hand".

At least in terms of crunching the possibility tree this should be a far less computationally intensive challenge.

What struck me looking at the AlphaGo papers was how human its algorithms were - it was basically doing what a highly orthadox human player would do (pick some moves that look interesting, read as deeply as possible) and then scaling up with a completely inhuman thoroughness and accuracy.

At this stage, AI wins in competitive games simply will not impress me. From here it may simply be a tour of force showing the breadth of fields AI can dominate in.

After that enthusiastic agreement; poker wins are evidence for the laypeople of something I suspect most people in AI research or game theory already know - a human bluffing isn't a magic advantage in a fair game. The AI can outperform on fundamentals.

The game of poker differs from Go and Chess in that poker players deal with incomplete information and therefore a successful strategy has to include randomness. If you are dealt AA under the gun, you can't just raise 100% of the time, you have to play a mixture of different moves, perhaps raise big 80% of the time, raise small 10% of the time and limp in 10% of the time (for example). With Go or Chess you can just play the best move in each particular situation and you will not lose.

It's just a different kind of game

Did I read the article incorrectly, or did it say that the bots creators were feeding it additional data each night before the next day's tournament? If so that defeats the entire purpose, doesn't it?

in what way would that defeat the purpose? the players can also review their days play during the night

Yeah... I guess I was just expecting the AI to be more self-learning and self-tuning...

It built most of its decision model by playing itself over "trillions of hands", according to Prof. Sandholm. You could almost think of the nightly reviews as a similar process to a human sleeping: we benefit from memory consolidation, pruning, time to reflect, etc. I don't see giving the computer 12 hours a night to do something close to reflection and self-analysis as "cheating," or even very different from what its opponents did over Mexican food each night.

The four humans were even "colluding" too.

This article http://spectrum.ieee.org/automaton/robotics/artificial-intel... says 120000 hands were played over the course of more than two weeks. How much of a factor was sheer boredom?

It was a factor for sure, especially as the AI would tank (stop and think) quite a bit at predictable spots and so the "Pros" would occasionally use strategies to try to knock out the AI before it would get to that point (they were shooting for a targeted number of hands, so maximizing the hands per hour is to the Pro's benefit). That said, they still had both money and pride on the line, and at least one Pro insisted that he was genuinely doing everything he could to try to win, despite the boredom and desire to be done with it.

Is 6-handed a significantly harder game or is it just (from the computer's point of view) just like playing 5 single games of heads-up?

It is much more difficult, as interactions between all of the players usually have to be modeled.

I would argue that it is significantly harder. In heads up, you can call a bet and see the flop and go from there. In a full table, calling early means you may have to fold to a raise. There is also a much more complicated analysis of the possibility that you are beat. Lots of times I may be in a hand and think I have the original raiser beat but when two other people call before me, the odds are much less that I have all of them beat.

I still think the hardest part of poker is the grind. The computer doesn't get bored or tired and play hands it doesn't have a reason too just because they are stuck in a dead streak. Taking a mostly conservative style, you'd expect the computer to out perform over time.

It would be curious to know what percentage of the time the players bluffed the AI successfully and vice versa.

No, the actions of players are not independent. For example, if 3 are in the hand and the 1st to act bets the 2nd is less likely to call because of the risk of the 3rd raising. Or was the 2nd only calling instead of raising because they expected the 3rd to raise based on recent aggression?!

It's a significantly harder game. Computing a Nash equilibrium in a 2-player game is relatively straightforward in technique (albeit very computationally expensive in the case of poker). There's several factors that make even 3-player games much more complicated than 2-players.

An additional player requires some sort of modeling of the interaction between players, which may not be feasible. For example, Player A may have multiple co-optimal strategies with respect to his own EV, but that affects the EVs for players B and C differently. Player A can choose arbitrarily between these strategies at any frequency, but players B and C can't predict this choice at all.

On top of that, Nash equilibrium requires that each player acts independently in their best interest. Teams can be modeled as a single player if necessary, but shifting alliances over the course of a series of hands can't be easily.

And on the pure computational complexity side, the state space explodes when you can have more than one opponent in a hand simultaneously. Combinations of players in a hand scales as n!, not n.

Hmm, why is it surprising that AI is good at poker? The way I see the game is that a bad poker player will just hold a model of his hand in mind. A slightly better one will also hold a model of his opponent's hand. Even better one will also model his opponent's model of himself... and so on recursively. And who's really good at recursion? Computers.

I tend to agree, but I suppose there is also a bluffing element.

How likely is it your opponent is bluffing, based on their past behavior?

Should you try and bluff this hand?

People have been trying and failing to do so for decades, the state space of the underlying incomplete information game is absolutely massive.

Put it against Negreanu.

"Oh you have AJ don't you?"

Negreanu wouldn't sit HU against the 4 players who played the bot.

Depending on the stakes, of course.

I'd watch that Twitch stream. (I've mentioned it elsewhere on here but I still find it strange that they were livestreaming and allowing the pros to interact with the viewing audience during the experiment.)

The University of Alberta bot has been beating top pros at limit hold'em for over 10 years.

no-limit and tournament play were deliberately placed outside the scope of their poker-bot projects--at least during the period of time i was following it which was approx. 2004 - 2010.

anyone know if the CMU team trained their rig on these variants?

EDIT: Read below, I am wrong. I clearly didn't know what I was talking about.

This is a great achievement in AI, don't get me wrong, but the headline should read, "AI beats the best four poker players we could find who were willing to play for a mere $200K".

All the actual best players play for millions and have a reputation to uphold. They would never agree to do this.

They four guys they got are pretty good, and could certainly destroy me, but they aren't the best of the best.

I'd love to see the bot play in the World Series of Poker for a few million.

my understanding is these are some of the best HU players the world has seen. WSOP is a completely different story being a super-large MTT.

>All the actual best players play for millions and have a reputation to uphold.

All these players are high stakes players. I think you're underestimating the fun factor. As for reputation. For a poker player having a reputation as being beatable is a profitable thing to have.

>but they aren't the best of the best.

Who do you think is? Like how many people do you think rank above this group at HUNL?

I agree, these probably aren't the best.

Same as in programming. There are some great programmers that we all know and that are public figures. But the absolute best? Probably making high 7 figures working in a dark room somewhere.

When it comes to HUNL specialists the best have to be public because there isn't as much action offline as in full ring and MTTs.

They aren't the best since I think that probably is safely in Polks hands but they are near the best.

Anybody here? http://pokerdb.thehendonmob.com/rankings/

Or any of these people: https://en.wikipedia.org/wiki/List_of_World_Series_of_Poker_...

HUNL is a different game to MTT which makes up almost all of the ranking values for hendonmob and the WSOP ME winners specialities.

Tournament earnings are utterly unimportant in the world of Heads Up No Limit cash game poker.

I like your posts, but you're misinformed on this topic. The pros selected are among the best HUNL players in the world right now. Offering the top winners in a separate variant not only misinterprets how different the games are but also falsely equivocates being the best with making the most money.

If someone asked you who the best mobile app developers were in the world, your list should comprise solely of mobile app developers.

You make a fair point. I clearly didn't know what I was talking about. I edited my original post to reflect that.

No worries, thanks for taking the time to listen. Have a great day

All of those people would get absolutely crushed at heads up no limit by any of the four players.

Jamie Gold won the WSOP, but he's a pretty bad player.

They are at worst top 15 HUNL players in the world.

How was this determined? Online play?

Yes from online play but there aren't any official rankings. It's hard to determine exact rankings because it's rare for 2 world class players to face off over a large enough number of hands to give a significant sample but a loose hierarchy does exist. You can tell where a player sits in that hierarchy by who he is willing or unwilling to play. It's not an exact science but I agree that all of these players are unquestionably elite and the AI has beaten them to an extent that it would be a very strong favourite over a team compromising of the true top 4 players.

Does anyone know the technical details of how to train and AI bot for a game like poker?

I imagine it's just reinforcement learning where the inputs are the actions of the individual players (hold/fold/raise, timing etc) and the statistical probabilities in terms of expected cards. Train a neural net to predict probability of the opponent's hands and act accordingly.

Is it just that the professionals all act similarly enough that the bot can learn based on other players?

I'm sure the actual process differs from research group to research group. Here's how one commercially available AI training system claims to work.

http://www.pokersnowie.com/about/technology-training.html

And they go into some of the problems that stem from how they've abstracted the game here:

http://www.pokersnowie.com/about/weaknesses.html

Sounds like the end of online poker is very near. This version required a supercomputer and took a long time to decide its actions but those type of things tend to be quickly improved given enough motivation.

Even if poker sites could somehow perfectly detect automated players(which they can't of course), highly skilled poker is profitable enough that some people would be willing to manually execute the actions themselves as directed by the AI.

Not really. 1v1 is just a small part of poker. Ring games are a whole different beast.

They are more complex for sure but not fundamentally different. I'm amazed the AI managed to win but now that it's happened it's bound to proliferate quickly because there's so much money on the line.

Online sites have been overrun by bots and cyborgs for years at all but the highest stakes. Most players are already being fleeced by computers online.

This article is related to no-limit heads-up, which requires a very different style of play than no-limit full ring. Full ring would be significantly more difficult to beat by AI. So the title leaves out some very important information. I am not surprised they beat heads-up; I would have expected it sooner.

There is a game humans can still beat machines at with ease, Diplomacy(1). When a machine wins a Diplomacy tournament I know we are finished as anything except pets.

1. https://en.wikipedia.org/wiki/Diplomacy_(game)

I'm yet to see a game of Diplomacy every actually end, so that makes it difficult. The only reason this game is difficult for AI is the social interaction required to play the game. If the computer was given a means of making offers and private communication with other players, it would be much like any other game. Rules, game state, probability, goal state, etc.

I'd really be interested in whether intentionally showing your cards instead of mucking them was taken into account by the AI. (e.g. showing your poor hand after a successful bluff)

It was not.

Does the computer-ness of the AI give it advantage in poker w.r.t. counting cards, at which humans are imperfect? Or are humans perfect enough these days?

There's no counting cards in poker. 52 card shuffled deck each time.

It's not "counting cards," but to be minimally successful at certain variants like seven card stud, I'd argue one must[0] remember cards that have been folded in the current hand.

But certainly in all HE variants (that I'm aware of), counting and remembering is unnecessary.

[0] It's necessary, but not sufficient.

Could someone explain how Poker is more challenging that games like Go? Please eli5.

I'm not qualified to give a comprehensive answer, but I would have to imagine that it partly has to do with the fact that poker is a game of partial information. In Go or Chess the current state of the game is known. In poker, you don't know what cards your opponent hold, so you have to assign probabilities based on previous actions, and simulate many different scenarios.

Another part has to do with the fact that both you and your opponent have surprisingly many legal "moves" at each turn. because not only must you decide to fold, call or bet, but if you bet, you also have to decide how much.

Does anyone have link to papers about the technology behind Libratus?

the endgame solver used is described in http://www.cs.cmu.edu/~noamb/papers/17-AAAI-Refinement.pdf

the first authors twitter account https://twitter.com/polynoamial/

Unfortunately this fails to impress me, merely because you could theoretically build a terrible AI and still win.

Just showing your ignorance. That margin of victory over that many hands isn't luck or coincidence. The pros were absolutely dominated.

I disagree and it's not ignorance on my part. Due to the nature of the poker and heavily weighted component of luck, a terrible AI can theoretically win.

I am sure the AI they built is a mighty and spectacular achievement, but poker is about the only game I know where the worst player can easily beat the best player. Prove me wrong and I'll happily accept your insult of "ignorance"

edit I should have been more clear, as I am extremely impressed with the AI's results of hands over time. I am referring to the context of a standard tournament where the loser is eliminated after losing their chips.

On a hand by hand basis, sure. In freezeout tournaments, yes. In heads-up over 120k hands? No. This was not a lucky win. And I say this speaking as a one-time full time player in a number of formats. Variance does not explain this loss.

"the worst" player might win a few hands, or even beat "the best" player due to luck.

They will not be able to consistently beat a better player, especially not professional players.

That's what this AI is doing. It didn't beat them once or twice. It beat them consistently over the course of 120,000 hands in a 20 day event.

Is it theoretically possible that "the worst" player could do that based entirely on luck? Sure, probably in the same realm as monkeys, typewriters, and Shakespeare.

In these contests they typically play each hand twice with the sides reversed to remove the luck factor.

So the deck for hand 1000 might be the same as hand 113853 but they swap who is the button.

Consider a game played in a series of rounds in which N% of rounds are decided by luck (random player wins) and (100-N)% are decided by skill (better player wins). As the number of rounds approaches infinity, the win percentage of the better player will approach (100-N/2)%

That is, if the game is 100% luck, the better player will have an expected win percentage of (100-100/2) == 50%

If the game is 0% luck, the better player will have an expected win percentage of (100-0/2) == 100%

If the game is 95% luck, the better player will have an expected win percentage of (100-95/2) == 52.5%

Conclusion: If there is any non-zero amount of skill in the game, the better player will win in the long run.

Yes it's possible, but it's also possible that a terrible chess AI could get lucky and win by the opponent getting a heart attack in the middle of a game.

> " I am referring to the context of a standard tournament where the loser is eliminated after losing their chips."

Why would you refer to that? That's not at all what happened here.

In the game that was played (long-term HUNL cash) the worst player cannot beat the best player at all. Not easily, not at all.

Not to mention they didn't run the cards out.

You're commenting without reading the article, seemingly.

The article mentions a similar tournament that occurred last year where the human players won against the AI. Accomplishing this turn-around in the span of a year seems like quick progress to me, and also impressive. It is certainly a link in the sequence to more autonomously adaptive AI. So, while what you say might be logically true, following the progress of AI would indicate that this specific outcome is, in fact, pretty cool. Whether or not you are impressed by it is up to you, but it is progress.

Surely nobody is surprised? Perfect knowledge of probabilities, no emotion and immune to bluffing.

Uh... that's a very dismissive comment for quite the breakthrough! Congrats to the human players and the AI team for a great showmatch.

"Capable of bluffing" would perhaps be the surprising thing.

To "bluff" in this case is defined as putting in more money than would be smart if all cards were played face-up. The AI is easily capable of bluffing and in fact should do so frequently.

It's not immune to bluffing, it's just better at it than humans are. Bluff too much, and your opponent can just call you whenever he has a strong hand, winning more often than not. Don't bluff enough, and your opponent can fold whenever you bet, knowing you probably do have a strong hand.

Find the right balance, and your opponent can't exploit you. With no-limit hold'em this is extremely complicated, and until recently the best humans have always beaten the best bots.

Nothing is immune to bluffing.

> immune to bluffing

There's no such thing.

An algorithm playing straight hand value based on probabilities is more susceptible to bluffing, not less. And this is no limit, where a single hand can swing all the chips.

Any poker AI that isn't a loser is going to have some pretty sophisticated modeling of the opponent.

Having a poker AI that plays deterministic strategies is an obviously terrible idea, for the precise reason you mention.

It makes much more sense for the strategy space to be the set of probability distributions over game moves (i.e. mixed strategies).

I think that the optimal mixed strategy for each hand is immune to bluffing (over many hands it will have larger expected winnings against a bluffer). If that wasn't the case, there would exist no Bayes-Nash equilibrium for the game, contradicting Nash's theorem.

It would be surprising to me if a completely deterministic strategy wasn't within some very close margin of an optimal strategy. For example, if you think you should raise with JJ 1/3 of the time and call the rest of the time in some spot, then you can raise JcJd and JhJs and call with the other jacks. This can give your opponent slightly more information about your range in some cases (for example, if they hold a J themselves), and of course you can only approximate correct frequencies, but it's highly unlikely (in my opinion at least) that such a deterministic strategy would be terrible.

Conversely, I'd be very surprised if any deterministic strategy were remotely good. I've played against people with reasonable tight-aggressive strategies who were a bit too deterministic. They get crushed.

It'd be far too easy to recognize when they have a good hand and fold and to push them off all their marginal hands.

Which really shows how the game has gotten so much harder. Ten years ago someone playing a fundamentally solid deterministic strategy could print money online. And in many low stakes casino games that is probably still true.

I was just making an informal remark, but reading your comment:

> I think that the optimal mixed strategy for each hand is immune to bluffing (over many hands it will have larger expected winnings against a bluffer). If that wasn't the case, there would exist no Bayes-Nash equilibrium for the game, contradicting Nash's theorem.

I believe that's true. I know for sure that heads up limit hold'em has been solved. That said, I think this context is similar to the iterated prisoners dilemma contest. There's certain to be an equilibrium, but what's interesting isn't the perfect strategy in a min/max sense, but rather a slightly suboptimal strategy that can detect and exploit suboptimal behavior in other players. It sounds like you know this area well, perhaps you can shed some light if I'm on the right hunch?

Yeah I think you're right. just playing an equilibrium is suboptimal in the sense that it doesn't extract maximal value from the opponent. You want to be playing a best response to the opponent's strategy.

That's what you see online poker players do. They model their opponents (in the sense of labeling them as fun player, too tight, too loose, etc), then try to predict their hands based on their moves. Otherwise I guess they would be losing money: poker is zero sum by its nature, and the casino's cut on top of that makes it negative sum!

Reply on news.ycombinator.com