Languages for online betting: an investigation of Go, Erlang, and Elixir
erlang-solutions.com
erlang-solutions.com
I really like when someone that doesn't have a horse in the race reviews some language (better if it has been used in anger). You gotta value the perspective of a pragmatic engineer that has misapplied a tool.
There's a lot to learn about where a language is best used, but when someone pay is somewhat tied to a language I don't trust the analysis as much.
Use the tools that get your job done and stop bragging about it.
I'm writing in C and decided to stay in a single OS thread and to instead write my own simplistic user thread scheduler. User thread ("green threads") work by simply switching call stacks and saving registers. In my case this is as simple as making a few Windows API calls like CreateFiber() or SwitchToFiber().
It's one of the best things I've ever done. Maybe 1 day of work to get the threads rolling, no bugs, and still in a familiar C environment. No bindings required to interface with the libraries that I need to interface with (like Qt) and perfect deterministic control over what happens (just not the other processes of course).
I think I can understand people are not using C doing web stuff (lots of strings) but for what I do I'm convinced it's the easiest, fastest to develop, most performant, most robust thing you can do.
All investment is just a page of code using the Windows Fiber API, and the scheduler ist just 10 lines of code doing round-robin scheduling. Which is totally fine right now, and there's the added benefit of being in total control and being able to make adjustments if they are needed in the future.
Since the scheduling is non-preemptive, threads are not allowed to block. But only a synchronous, blocking coding style is maintainable, so threads use wrapper functions to execute slow operations. The wrapper functions have a synchronous interface, but internally they call asynchronous functions, and unless the request can be served immediately, they call the scheduler. The wrapper functions make another 50 lines (so far) of really simple bullet-proof maintainable code.
It's really like Javascript technically (non-preemptive scheduling), but you don't have to do callback chaining. You just write your code in an ordinary synchronous blocking style. Think about that. You write thread-heavy interactive C code that is much more maintainable than Javascript code. And of course, much more maintainable than clumsy OS threads - no management, no synchronisation primitives and no excessive copying between threads. On top of that, the scheduling is deterministic. All with 100 lines of straightforward infrastructure code.
The only boilerplate in the project comes from using Qt/Qml.
(If you're on POSIX and can't use the Windows Fiber API, you can depend on the deprecated makecontext/getcontext/setcontext API, or write a little inline assembly. It's not a big deal either way, since it's really just a few lines of code that are neatly separated from the rest of the system).
This makes me wonder why Erlang/ Elixir are used so little in the financial industry? I have been working for a couple of financial firms and even a HFT company but I have never seen Erlang being used, despite its apparent advantages.
Does anybody know why this is? Or is Erlang actually being used a lot in finance and have I just been working in the wrong places?
Hiring for Erlang can be difficult too.
That being said, Erlang is used heavily in FSI outside of HFT, or algorithmic trading. Several large FSI institutions are known users of Erlang (Visa, Banks).
Let's say that there is a football game Liverpool vs Manchester.
One client puts £5 that liverpool will win. The exchange takes the money and adds it to the order book.
Another client puts £5 that manchester will win. The exchange takes the money. It's matched with the opposite order and frozen. The money will be redistributed to the winner at the end of the game, minus a fee.
It's exactly like buying and selling in a normal exchange. You can do real time trading during the game and arbitrage between exchanges.
Disclaimer: I worked in betting before I moved to finance.
Erlang is super weird and hard to recruit for. When used right, it's not faster or more productive than C++ or Java. I wouldn't advise anyone to use Erlang for anything.
Try concurrency on multiple cores in C++ or Java, and let us know how you get on.
I'm quite happy and doing well with my 'super weird' secret weapons, thank you!
#pragma omp parallel for
I wrote C++ and java for a living. I can get on just fine, thank you.
Certainly, well-written C++, and often Java, will be faster than Erlang. Erlang is never going to be the fastest language around, that's not why it exists.
But more productive? It takes significantly less code in Erlang to achieve most tasks in its wheelhouse than C++ or Java. I wish I could remember the numbers, but when I did a very rough LoC comparison of Cassandra vs Riak, I believe Riak was roughly 10% the size of Cassandra.
What about payoffs and probabilities? A financial firm can buy/sell in open market (often with market making privileges) to remain delta neutral but what does a betting exchange do?
A user bets on a result of an event, with 20% probability of the event happening. This means that someone (often a market maker) is willing to bet with 80% probability of that same event not happening. User waged £10, and if they win, expect to net £40. That "other" £40 came from the counterparty.
A betting exchange facilitates this. It sets aside ("escrows") the funds from both parties' accounts, and once the results of the event are known, settles the trades. Funds from escrow, minus the commission, are credited to winner's account.
The basic premise really is that simple. Now, running an exchange is anything BUT simple.
In a normal exchange, you see the price and it goes up and down. In betting, you see the probability going up and down. It's also the multiplier on what you will win.
In the previous example, they would bet £5 at 50/50.
Erlang has OTP, Go does not. Erlang is FP, Go is not. These are the critical differences.
Silver lining (in context of OTP) for Go is that the du jour 'cloud' based architectures will provide the OTP like capabilities ("elastic", "fail-over", etc.) for deployed (micro-)services. It seems that in the (future) long-term, Go is better positioned than Erlang for cloud based systems.
I mean would you like to bet on that stack on top of your betting platform? >___>
Pony promise great things and the language have a proof and all that but it still needs to be battle tested and get to some sort of stability.
https://www.ponylang.org/discover/#what-is-pony (under why not pony section, API unstable)
The degree of tech hype and adoption of bleeding edge is crazy. The thought of programmers just betting on these thing without thoughts on the schmuck who would inherit these technical debt still boggle my mind.
1. Highly domain-specific legacy pricing model running on... Excel spreadsheets. These present huge performance bottle-necks and aren't easy to migrate.
2. Reliance on 3rd Party data. Did a scout just send incorrect data? Did their device disconnect? Did they rollback the wrong information? Is the source lagging behind the TV/online feed? GGs. Suspend the game and refund bets.
The tooling really isn't that important. Scalable software can be written in any software, and while some languages do make things simpler, often you can just also find a suitable paradigm (or middleware) that can provide similar advantages. Having said that, we do make extensive use of RabbitMQ (which is written in Erlang) in our environment. I suppose this is also just an example of my previous point though...
The only issue is with low popularity events. Noone cares about them, there is low liquidity and volume, it's hard to price them.
Yes, exchanges support real-time trading, but not every company has all their models running in modern paradigms. It's not uncommon to find companies that run multiple VMs for the sole purpose of... running Excel calculations. Yes, this is exactly what it sounds like. A server that has a several, multi-meg Excel files in memory with external APIs that plug values into cells and read outputs. As I said, within the grand scheme of things, for sports who still run on legacy models, this is a major bottleneck. Remember, many of these companies were started in the 90s, and that's when these models were created. The people who created them are usually swallowed up by the financial industry and now come with a hefty price tag. So the question for the business is "Do we spend millions recreating these proven models and run the risk of them failing? Or do we simply just add another VM?"
At my company things have slowly started changing, but the new models (written in F#) still need a lot of work.
I agree that the finance industry is competing for talents of quants and developers. Anyone who works in betting will find out that finance has better work conditions and better pays.
Yep, got it: it was all done in PL/SQL. Clicks on the website went through a simple Java layer and called stored procs. At the time we were doing more transactions/sec than all the European stock exchanges combined, and often the Japanese too. It makes me chuckle when people say to me now "but SQL databases don't scale" when they are running 1/10th of the workload we were running 10 yeara ago...
No idea what they're using now, mind, I left after the IPO but before the merger.
No benchmarks, no numbers, no real detail, just empty praise for Erlang from a company called "Erlang Solutions".
"When you pit Go’s requests per seconds against the likes of Ruby on Rails or Django, Go returns some impressive benchmarks performing 3x better. Go can scale to hundreds of thousands with relative ease, in much the same way that Erlang and Elixir scale to millions."
He's talking about raw performance where Go is much faster than Erlang and still talking about different order of magnitude to diminish Go.