ASX drops plan to replace CHESS with blockchain, writes off AUD $250M
insideadviser.com.au
insideadviser.com.au
I've skim read through it, and it reads vastly too weak.
They used a blockchain haskell implementation on a fringe blockchain technology, and wrote nearly all their business logic in it.
If I were them, I'd immediately sue whoever built it. This level of gross incompetence reads to me like they were scammed, used as patsies to integrate blockchain into a client.
It bends over backwards to say that somehow this approach meets some non-functional reqs. But it cannot plausibly meet any such requirements well. Even praising the "quality" of the haskell code whose job it is to provide a casio-watch level of compute power over an immutable distributed ledger. For $250m!
It would be fascinating if, eg., the Australian gov forced Accenture to release their internal docs. I imagine there are several versions of me with with much much more strongly-worded things to say.
This feels like it would go well in a netflix documentary retrospective on the "blockchain con" and how it duped gen-z and ceos.
Or if you have to pay 100mil for the drop in already complete solution (which sounds insanely expensive).
Much of what board members do is play games to keep themselves in power.
The most likely to work solution - rip out everything in the company and start from scratch. Short of that you are looking at - $50mil to modernise with a drop in (fully functioning) component in a 2 week install + 5 years and $100mil to write translation layers to all the other systems which will still fail on edge cases.
Every now and then the executive that believes in sunshine, rainbows and unicorns comes along, slaps down a stupid budget and tries, with good intentions I might add. They however, are predisposed to buy into the people and vendor that sell magic beans. And they cycle continues...
1. Those made the decisions are smart, really really smart people. If they were dumb, there is no way they can get to where they are now. So if the decisions look questionable, they probably are indeed questionable. But that does not mean they made a mistake. Smart people don't do obvious stupid moves, if they do, there must be some reasons we don't know.
2. If the budget is not used, at best it lapses or worse will be used by someone else. So do whatever they can if they have nothing better to do with it.
3. If you do something for someone-else with someone-else's money, neither the price nor the quality would be much of your concern.
4. After handling meat, you know, there is oil on your hands.
Put all that together...
CHESS Replacement Application code consists of Daml, Scala, and Haskell like Daml with ~87% of the code in Daml/Haskell like Daml and 17% in Scala.
So not actually Haskell.
Source: I was the evaluator of DAML for Jpmorgan.
The way daml is setup makes it more of a weird sibling of GRPC for stateful resource protocols. But where you have to write your own db and interpreter tooling to ingest the commands if you want all the usual nice things you’d expect from a database
> The way daml is setup makes it more of a weird sibling of GRPC for stateful resource protocols.
Yes, well said. I think that's quite fair. It's wrong to look at it through the lens of Haskell (or any other FP language), it's really a declarative language for describing process flow and actor rights and obligations. Not something you could write a complete program in. A better way to think of it is as a big DSL for controlling state machines; you still need all the machinery around the outside to advance execution, handle IO, trigger events and transitions etc.
It's syntactically similar to Haskell because the people who created it were Haskell heads out of ETH Zürich. That's the only connection.
I’m not sure if I was the most patient of evaluators ;). If I was despite that one of the more patient evaluators, wow!
I kinda view everything in this space from the lens of tools I want to build for myself. Namely a decent transactional db system where the modelling language for datatypes and workflows is a linear logical concurrent functional language with some soundness guarantees. I did build a preliminary version of the core of that at jpmorgan, though for various reasons it never saw the light of day
That's right. It happened fairly quickly, part of the efforts to formally verify the language. I think when you were poking at it was around the time it was changing underneath. But, that was years ago!
> Namely a decent transactional db system where the modelling language for datatypes and workflows is a linear logical concurrent functional language with some soundness guarantees.
I mean that's largely what Daml was trying to do. But of course it lacked the transactional DB system once we tossed out our in-house DLT. The approved answer back then was, deploy it over someone else's blockchain/DLT. Well, we all know about those. So before I moved on I spun up a successful skunkworks effort to integrated it into some relational databases; you'd be surprised what _that_ implementation ended up at the heart of.
Well, patient is maybe not the right word :) You weren't ever a dick though and that put you in a small club. You can draw your own conclusions about the space I'm talking about (Haskellers, JPMorgan, enterprise DLT...)
(I'll ping you separately as I have you at a disadvantage. For various reasons I won't put my name in this thread.)
I was using it in the sense that Daml is "a haskell", in the way that Ocaml is an ML, etc.
And
“ From a participant standpoint in the current design and architecture (not withstanding future use cases), there is little value to processing all the business logic on-ledger as ASX maintain data integrity as the market operator and participants receive a point-in-time view via API contracts.”
In this case, $250m seems to me a reasonable sum to ensure that the ASX will never again get tangled up in crypto pipedreams.
A blockchain solution can make settlements faster, final and prevent the current issues. It also means you no longer need a broker to hold your stocks/assets. More sophisticated users can run their own "wallet".
No wonder exchanges and consultancy firms are getting away with baffling the public and businesses with bullshit.
how blockchain would block theft without blocking valid transfer? This requires checking real-world data so all blockchain stuff does not help at all.
(the same transfer may be considered as valid or as a theft by different groups of people! See various stories about evacuated gold reserves of countries that were taken over.)
Basically you can't force everyone to do transactions on chain. The benefit you mentioned only exists if everyone is on chain.
You can create derivatives of whatever you want, it does not matter how the underlying is delivered or settled.
And the settlement price could not be faked but you need an exchange that offers bid/ask prices and that matches the transactions and that could clearly be faked.
“Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, one by one.” - Charles MacKay, Extraordinary Popular Delusions and the Madness of Crowds
Your scenario is you can’t trust your stock exchange to correctly known who owns what stock, and you cannot trust their data feeds, given the heavy and necessary regulation of stock markets, given the very goal of them allowing ordinary people to invest in the largest companies without DD in a fungible way.
In that scenario a stock exchange becomes useless regardless of the technology.
Blockchain has other problems too. It is slow! It can’t keep up with the source of truth. And it cannot be the source of truth without slowing operations down massively.
For blockchain to be worth it you would need different validation nodes operated by different entities and consensus. This will mean the end of fast fulfilment of trades most likely.
If your point was about simple api bugs then that is an interesting point. Decent operations and monitoring should prevent this. I would put data loss or accuracy bugs with stock exchange data from the source in the same bucket as “making the exchange useless” as it is not providing a fair system.
Even in the US, there's a lot of bs that goes on related to high frequency trading. With a trustless, decentralized system, the playing field could be evened (instead of favoring those with LAN connections to the datacenter). And who knows in general how much order book bs is happening behind our backs.
And the blockchain being slow isn't really that much of an issue. Stock trading is already settled at a later time, so stocks being settled to blockchains at a later time, batched, wouldn't be a problem (and real time trading would happen with L2 or L3s).
Some company was parasitically using them to drum-up interest in this useless product area.
Before they fixed that they wanted to move CHESS to a blockchain. Wild.
The project got the underlying Blockchain tech from VMware... AND had a requirement for a backup "traditional" RDBMS system.
So, complete feature duplication of the core, mission critical application.
Scope creep was crazy. Someone who doesn't understand the technology went "blockchain yeah sure mate let's go."
Good project managers are so important.
Maybe product director / VP or whatever
That's more an issue with Product Management.
A requirement likely put in place by someone who realized that even if executed properly, a blockchain solution would do no better than what the "legacy" RDBMS already could.
Really the ASX should just focus on providing a solid modern platform for trading. Simplicity, security, and speed. None of that requires any form of blockchain which is slower, scammier, and more complex. It just requires writing a modern electronic computerised system.
http://www.bom.gov.au/
Maybe there's a technical reason, you're about to say. Maybe they have an enormous load on their systems and they can't afford the overhead of TLS? Or something?
Get a load of this: https://www.bom.gov.au/ works!
It shows a full-page error message -- via correctly configured HTTPS -- that says:
The Bureau of Meteorology website does not currently support connections via HTTPS.
You will shortly be redirected to http://www.bom.gov.au.
Their excuse for this was that there already was an ongoing project by some consultancy (Accenture or Deloitte, I can't remember) to uplift their site. Years later that seems to have gone nowhere despite tens of millions of dollars spent.
Australian organisations seem to love this kind of thing. Still clinging on to the old system for dear life while burning a huge pile of money that doesn't actually deliver any improvements and gets abandoned after years...
Also, I can't select a location manually, it forces the use of GPS location finding. Even if you bookmark a specific location and then load it in a new tab, it'll immediately prompt to use your location. If you refuse...
Unable to detect location.
To use your current location, allow permission via browser settings.
I bet this page cost millions. It hides the information I'm looking for "below the fold". It shows a grand total of 276 bytes of readable text on an entire page of HTML that took 198 requests, 2.32 MB and 19 seconds to load.
As I mentioned, I have to scroll down to see the content I actually wanted, which is the hourly forecast. But despite making you scroll, it still hides half the content with a "show more" expanding section you have to manually click. The "saved" area is a tiny, tiny fraction of the multi-page screen real estate they waste on enormous empty space.
The radar map no longer shows temperature measurements or anything useful, and is staggeringly slow. It stutters my very high-end gaming laptop. This is in part because BoM was too incompetent to host ESRI ArcGIS themselves and used the cloud-hosted "tiles.arcgis.com", which is hosted in the United States, 230 milliseconds away from Australian users.
Speaking of United States data centers used for Australian Government websites, I've noticed that it seems to be the policy of both state and federal departments to add the Google Analytics to all pages to ensure that Google can track our citizens during their interactions with their government...
Someone came along and gave me that site. Bugger if I know how to change it. Once it's gone it's gone!
On the one hand, https on BOM might not be a big deal / low priority in the grand scheme of things, but in the other hand it could be done in like half a day by anyone there who knows computers.
It's probably safe to say they don't know how to computer
The government just needs to hire coders directly, that's what Service NSW does and it shows.
Lots of businesses are now tech companies, but the managers are from a different age so you need layers of other managers to do exactly the same job (when people talk about weak productivity growth from technology, this is it...it took half a decade plus after the industrial revolution for things to filter through, it will be a bit quicker this time but there are massive institutional roadblocks...perhaps even bigger than in the 19th century which was all de novo innovation).
There are a bunch of potential projects I could knock over personally in a month or two each that I will never ever be allowed to bid on.
Instead, it goes through an internal process at the department that takes months, which means a bunch of people have to justify their positions, driving up the minimum cost.
These non-technical people architect the solution, put it a scribble of a Frankenstein's monster of a design in the tender with a laundry list of requirements, none of which are actually needed because what they really wanted was a mostly static website any idiot could develop.
A couple of years later (I wish I was kidding), the tender process is finished and a Big Consultancy begins the hard work of... subcontracting the work. To my parent company, that finally gives it to me. Out of the $millions on the tender, I now have 2 weeks to a job that should have been 2 months.
The design is now hilariously out-of-date, and was wrong to begin with, but it is now a part of a legally binding contract between orgs I have zero leverage with. It MUST be implemented, verbatim. Including the bits that are actively harmful, stupid, backwards, legacy, or just idiotic. All of it. This stuff is audited to make sure there's no corruption or dodgy work!
The end result is a monstrosity built in a hurry by a poorly paid tech, while 10+ middle manager types all take their cut.
It's abhorrent.
The only sensible reason I can think that BOM is not under https is that there are ancient mission critical consuming services somewhere that can't handle the upgrade.
For example: a network of regional flood warning alarms that are pulling flood/rainfall data from a feed on the same url. They could have been built in the early 80s, and cant be upgraded. Ignoring them would risk lives etc etc
Nevertheless there would be other ways to solve that.
There. I’ve solved it.
Similarly, a filter based on User Agent would likely also work.
That's purposeful incompetence, not an accident, oversight, lack of budget, or anything else that can be excused.
Even if there was an ancient service that can't handle HTTP, keeping HTTP and adding HTTPS support will do just fine.
My guess is that their entire system is so tightly coupled that they can't unpick this without a huge amount of work that requires real developers to stick with it and get around it.
London might be seen as conservative/cautious compared to SV or Austin or Lisbon, but it's far ahead of Australia where bloated big bang projects are well supported, and lean projects by actual tech companies are ignored.
However, holders can’t opt themselves in; inexplicably this needs to be supported at the broker level. I’ve spoken to one of the large brokers who advised me they have no plans to support this feature.
How the smeg does an MVP or PoC for something like this even cost $250M?
Also if there's a new CEO they are certainly going to want to clear out the cupboards, hence the call to Accenture (doubtless with some significant guidance on what the client expected in the report).
This is why continual delivery is preferable over a big bang. What could you blockchainify for a million. Then the lesson that blockchain is crap is cheaper.
Where are your small incremental milestones? How do you deliver working software every two weeks?
This was just a big, hard, database project. It's not a "textbook" easy to deliver project.
You just need smart cookie project managers and Chief Engineers to manage whether you do 1 week milestones or 1 month milestones.
If what you are doing is very novel those deliveries may be proof or disproof of concepts. The customer being yourself in a way.
In 6 weeks it should be possible to deliver “does blockchain make sense”.
The question is if the answer is NO does the org have the guts to can it. Someones paycheck might depend on keeping it going.
Sounds like the problem was mismanagement. Let's not blame crypto for everything.
I'm not saying blockchain was the answer to their problems, but it could have been a sub $10M mistake if not for the terrible management.
Here's the Accenture report in full published by ASX: https://www2.asx.com.au/content/dam/asx/markets/clearing-and...
They failed to architect the project properly by trying to run everything on the blockchain smart contracts instead of leveraging something off chain. Similarly there was mismanagement from DA (the developers) and ASX with no clear goal of what the end product looked like or how progress was being made.
Accenture even call out that a blockchain solution might not have been the best one considering the ASX centrally settles trades anyway.
The pdf is a good read into the technical and management failings of the project
Much harder to convince anyone to do Blockchain work now. It's a blow to Blockchain's reputation.
The ASX brought 5.4% of digital asset and from what I heard, they were a core part of building the system. https://www.ledgerinsights.com/asx-150-million-in-blockchain...
Does anyone actually believe the level of precision coming from this report? I bet good money that remaining “37%” would take way longer to complete than that supposed “63%” did.
Oh, Accenture created the report, so it must be accurate, never mind. /s
I've found that when communicating with "leadership" it's best to not present anything difficult that requires them to make judgement, because they'll use that as an excuse to do what they wanted to do anyway (60±20% that's basically 80% and there's always some uncertainties with these things, so that's basically 90% so it's basically done, just ship it).
Providing uncertainty is just asking someone else up the chain to pick the number. It's better to just pick it yourself to shape the conclusion you're going to make. The people making these decisions aren't scientists. They don't have any allegiance to your methodology or analysis. They just care about the conclusion.
"Unfortunately, ASX management and its board of directors have proven over the past seven years they do not have the technical skills to manage a large-scale technology project or the self-awareness to know that what they are doing is investing in a technology dead end."
https://12ft.io/proxy?q=https%3A%2F%2Fwww.afr.com%2Fchanticl...
"They were building a Frankenstein’: how ASX’s blockchain unravelled" https://12ft.io/proxy?q=https%3A%2F%2Fwww.afr.com%2Fcompanie...
Some people are blaming "tech nerds" for having too much influence over the project. Really the problem was charlatans. If they had hired good tech people the blockchain concept would have been binned years ago.
Another win for the Australian tech scene. Optus, Medibank Private, ASX... just embarrassing.
I've seen places which will rather hire 10 developers getting paid $80k a year than two developers getting paid $400k a year. You get what you pay for. The number of managers who will not hire someone getting paid more than them is astonishing.
1. Much less likely to challenge them.
2. Numbers look much better.
3. When things went south, there would be someone to blame.
4. They are not great at assessing and contextualizing skill and experience
Additionally, with good leadership 10 x $80k would almost certainly outperform 2 x $400k developers.
There are lots of complex failure modes in a system like this, and minimizing them is an advanced skill. The learning curve for a competent generalist senior engineer to get good at these kinds of problems is significant. Designing and implementing a national securities exchange is a big deal.
This is a place where you want either experienced developers who have worked on other stock exchanges, or at least FAANG engineers who've worked on systems with analogous challenges. This is not a place you want to skimp on salary.
Of course OP thinks that $80k is a good salary for a developer so that ought to tell you how competent she is.
Individual engineers are not necessarily talkative types.
They don't necessarily have the ability to influence feature creep in the boardroom.
The other developer and I just looked at each other and wondered quietly if the consultants knew they were talking gibberish.
So many large Australian institutions have been captured by consultants, snake oil peddelers and grifters, all with their hands in the jar. It’s embarrassing.
Time and time again, Australian non-tech companies cut technology jobs to cost cut.
Anyone competent looking to build a long term career working in the corporate sector either moves to sales/marketing. Or, these days, moves to Atlassian/Canva.
Over a rolling 20 year period, the process self selects for the least able.
The IQ at the "Head of Engineering" level in Australian non-tech is minuscule. Selection bias.
Most of them don't have CTOs, they have CIOs who report to an accountant in the C-suite.
All the offshoring and outsourcing (and cultural disdain for technical people) means they can no longer tell the difference between charlatans and good tech leaders. But the good people know how to avoid them.
It's not clear to me whether the $A250M was the direct costs to the Exchange (ie invoices + ASX staff time) or the value which the Exchange was carrying the software on their books at ?
They are the same thing. The wage, service and material costs through development are capitalised as an intangible asset. (Essentially the attributable development costs become the value of the asset.) This intangible asset just got written down to zero as it was accessed as having zero future economic return.
As I say I'm not an accountant ;-)
That's why the majority of crypto is bought and sold at crypto exchanges rather than on the blockchain.
All other hypes are dead. 20 years.
You can run a ridiculously large business off a monolithic RDBMS.
https://www2.asx.com.au/about/regulation/clearing-and-settle...
https://www.commsec.com.au/support/help-centre/About-the-sha...
The settlement cycle for the NYSE is two business days after the day the order executes, or T+2 NADAQ T+2 settlement cycle, etc. etc. etc.
Thats how it is meant to be.
They should have used blockchain - put your daily status and reporting on a blockchain and have it visible to everyone in a decentralized manner. No more silos or misaligned views. This is one of the many problems that blockchain solves.
Time and time again we see the same result in projects costing taxpayers or shareholders millions. There seems to be no shortage of those (at every level) ready to believe that one can just apply some “technologies” to very complex legacy systems that will result in something new and effective.
Expect Accenture and those peers to be first in line for slices of future ASX pies.
And this isn't the first time. A few years ago, ASX ran a massive project to replace the matching engines for all their exchanges. After burning a large pile of money, it too blew up, and they abandoned the whole idea.
But everyone involved fails up and out with their careers and bonuses intact, so ... she'll be right, mate. No worries.
I wonder how they came up with this oddly specific number.
Seems like the delivery partners were:
* Digital Asset’s Daml smart contract development capability
* VMware’s DLT (blockchain) platform
Does anyone have any information about those?
https://blog.digitalasset.com/press-release/asx-digital-asse...
I wonder if ASX was project managing this internally or just throwing poorly thought out requirements over the fence.
With any luck, the replacement program for the replacement program is a little more focused on reality, but ... I wouldn't hold my breath.
JPM use of their own Onyx blockchain makes the repo market far more efficient: https://www.bloomberg.com/news/articles/2020-12-10/jpmorgan-...
Visa B2B Connect for global payments: https://developer.visa.com/capabilities/vba
I get it, blockchain makes for a flashy clickbait headline. There are bigger problems in the project than the selection of blockchain as a solution.
- Project was 63% with an uncertain timeline to completion. That alone is a red flag for any enterprise project.
- The teams involved weren't communicating with each other resulting in confusion. aka "misaligned views".
- When it comes to scalability, there is no way to rate limit participants, so they could screw the system given it couldn't process more than 100k transactions (not sure what rate since it is unspecified).
There are multitude of issues that justify canning the project.
For those unaware ASX used VMWare's DLT blockchain: https://www.vmware.com/welcome/asx.html. There's nothing "scammy" about it.
I'm unsure of why people in the comments here are still confused between cryptocurrency and blockchain especially since they have distinct names. If you're gonna point out incompetence, you have to realize your own faults as well.
Anyone who claims to have enough information to make a determination about blockchain tech being the cripple here doesn't have the information required to make it given the problems pointed out. There's plenty of financial industry users making use of it already.
Lots of tunnel vision below, but that's unsurprising given the hate against crypto (which people below often confuse with blockchain). I remember the same bias against K/V dbs (NoSQL). People would blame the tech choice rather than the circumstances. It's easier for certain types to blame tech for everything then hail it when the project succeeds even though it was only one part of the equation.
This was one of those times.
Do that 250 * AUD/GBP rate times.
edit: shit forgot to adjust for 28 years of inflation
This is in my opinion rather a good sign instead of a red flag:
If one is honest, nearly every enterprise software project is something like this. It is rather that nearly all managers are not honest, but pretend that everything is under control and tell you of some fictional timeline for completion.
The fact that they openly admit the state and the uncertainty to completion is to me thus rather a good sign that the stakeholder are much more honest than is common in such projects.
If you see this from an CSci (data structures & algs) pov, you can quickly see it's a extremely niche academic solution to a problem almost no one outside academia could plausibly be interested in.
A blockchain is not a "valid solution for fast..." anything. It's practically useless machine-trust guarantees impose impractical overheads for almost every basic operation.
Any one claiming blockchain is a plausible solution to almost any technical problem is profoundly mislead.
Now, have they? I was surprised to hear this because I was under the impression that blockchain is generally useless in an environment where the participants can trust each other. I searched for "visa jp morgan blockchain" and found these:
https://www.forbes.com/sites/emilymason/2022/10/11/jp-morgan...
https://www.finextra.com/newsarticle/41102/jpmorgan-and-visa...
It seems that JP Morgan is offering a service called "Confirm" to validate account numbers, which Visa is using. This service is apparently offered on a blockchain. But nowhere does it say that these companies would be using a blockchain to actually settle real transactions. And it indeed seems a service to validate account numbers could just as well be offered with a more traditional API, like HTTP/JSON, instead of blockchain.