Edit: I feel like we need somebody like Coffeezilla to unwind the web3 mystery in a digestible format for the rest of the tech world.
Seriously web3 is just another way to create hype for arbitrary crypto tokens and Ethereum. In a few years there will be something else - quantum web or something to suck new buyers in.
I have very hard time imagining anyone working in web3 with good intentions.
1. Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees.
2. This service is better than other web3 api providers.
FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me.
If it helps, you can remove web3 entirely from the problem to understand it. The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. The other main thing is that there is a standard API that people provide (read: sell) access to, which can be slow and/or expensive if you've got to pull a lot of data from it.
So, Apache Kafka.
Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it
> so you can build something neater because you've got guarantees that many other problems don't have.
So, first you have to find problems for your solution
For services that are using the "append-only log as your source of data" then you need a tool that manages the "expensive to reconstruct it" side of things. I think that is what this does: it materializes and abstracts interacting with the reconstructed view of the data.
Does that give access to the full history quickly?
> Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it
Yeah I don't want to do that, but I do need to in order to process the data. For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
It doesn't because no append-only log does. Apache Kafka is that "unique web3-only thing never seen before in traditional systems" that no only has been seen in traditional systems, but has been successfully used in those systems at scale dwarfing all of crypto, and has well-understood behaviours and patterns.
Edit: In 2019 Linkedin (who developed Kafka) were processing 7 trillion messages a day: https://engineering.linkedin.com/blog/2019/apache-kafka-tril...
Edit: In 2018 Pinterest was handling 15 million messages a day with 1.2 petabytes of data per day https://medium.com/pinterest-engineering/how-pinterest-runs-...
A view of the final state of data is literally one of the things they require, and everyone knows how to make one.
> For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
So this service just creates a "traditional" database with a "traditional" access method (actually an ancient access method, RPC) with a "traditional" ElasticSearch on top and pretends it's something new, never made at scale, and something no "traditional system" was ever prepared for.
Edit: see LinkedIn and Pinterest above. Or Facebook. Or Amazon. Or AliExpress. Or... "Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie that I can't imagine people who can look at it and say "true, true". The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
Then it doesn't solve the problem people in the space have. I understand the visceral hate for web3, I do, but it really gets in the way of talking about a very bland technical issue.
I think you're misunderstanding what this service is. It's about access to Blockchain data not about running a chain.
> A view of the final state of data is literally one of the things they require, and everyone knows how to make one.
That's one thing it needs, but only one thing. Keeping up to date with this is a problem of monitoring mostly to keep on the right side of any forks. It's not a data problem, but that's not what the API is providing.
> Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie
It totally would be, but it's not what the site said. It's literally just saying that they're doing better than off the shelf tools because of guarantees most other problems don't have. And I'm not surprised, the data isn't of a crazy scale though is a bit annoying, access patterns are extremely predictable and depending on which APIs you're supporting building a specific thing for it is fairly straightforward.
> The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
Nobody is talking about ingest speed really. Unless you're also applying all the state transitions because that's quite annoying.
=== start quote ===
Web2 Databases were meant to deal with datasets that change frequently. They have to provide ACID guarantees to ensure correctness.
The solution to this problem becomes more complex with scale. Distributed transactions, sharding, replication, and fault tolerance are notoriously slow.
Web2 databases are just NOT designed for web3
A system designed to take full advantage of the fact that blockchain data is immutable, and correctness has been dealt with already.
We built a cheaper and simpler storage and indexing system from scratch that can scale to the heaviest blockchain workloads out there.
=== end quote ===
Then here's what you claimed
=== start quote ===
Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees.
The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have.
=== end quote ===
So, to give you a doze of reality and dispel the lies on the web site I showed that your view of the world does not reflect what the world is:
- Blockchain isn't the only thing to have an immutable log. Apache Kafka is one, and is being used at scale that dwarfs anything in crypto space
- "The heaviest blockchain workloads out there" as claimed by the web site have literally nothing on "traditional/classical" whatever because traditional/classical handle workloads that exceed blockchains by multiple orders of magnitude
- There's nothing revolutionary in the service despite its bullshit claims: they dump blockchain data into a traditional database, slap a traditional (I'd say conservative) JSON-RPC API on top, and probably add a traditional Elastic Search on the side. Everything that they accuse "traditional systems" incapable of doing and incapable of handling. Even though, again, traditional systems can probably handle all this on a free tier of any cloud hosting provider
We've known how to handle append-only mostly-immutable data at scale using "traditional" methods for ages. Literally every banking system is all but required to have an append-only log of transactions for audit purposes. And yet here we are with clueless gullible fools pretending this is something new and something that can't be handled by "outdated slow technologies of yore".
They don't say that standard databases can't handle this. They say that they can do it cheaper and faster without them, because you don't need a bunch of the features. They are literally telling you that their solution is simpler not revolutionary.
You are conflating two very different things, the production and indexing - and then consumers.
> - Blockchain isn't the only thing to have an immutable log. Apache Kafka is one, and is being used at scale that dwarfs anything in crypto space
Sure, and doesn't solve the problem here as you have already told me. Kafka would be ridiculous overkill IMO for the part it would play. The problem is simpler than kafka is designed for, so it makes sense that someone could make a simpler tool to handle it.
> Even though, again, traditional systems can probably handle all this on a free tier of any cloud hosting provider
That very much depends on your users - it's this kind of statement that makes me think you're not picturing what this actually is.
> slap a traditional (I'd say conservative) JSON-RPC API on top,
Yes, this exact API is very much a requirement for this kind of service.
As a final thought, if you think you can throw this together for a large customer base easily and host it basically for free while having better performance - go for it. You'll make money because the alternative options are expensive.
But since this was your last reply, adieu.
"Web3" doesn't lend anything to technical conversations and is best used on board rooms full of coked up investors interested in dropping money on another dumb crypto bro's vision to change the world.
Even so, in a technical setting I often hear engineers refers to "hosted function" or "function service" when referring to "serverless" infrastructure.
Nah. Blockchains are (of course?) not the first database use case where immutability is necessary. Blockchains aren't unique in terms of access patterns, either.
Append only databases have been around for decades.
Heck Kafka is an obscenely popular example today.
I've talked to people working in fields with legally mandated immutable databases, again decades old technology. In the most extreme cases the immutability is enforced using technology, such as a storage medium that only supports writing and not deleting or changing data once it has been written.
Distributed append only databases based on not trusting the people you are doing business with, and where writes cost small fortune, sure, that is new.
But the # times I want to make large transactions where:
1. I don't trust the other party 2. I can't use one of the insanely well trusted escrow agencies, some of whom have been around for literally hundreds of years and have built up trust accordingly 3. I am OK with the transaction not being reversible through the typical legal system remedies
is miniscule.
The idea of a distributed ownership ledger is more interesting, but again, if you have any level of trust with your partners, it is cheaper to throw away the entire proof of X system and replace it with a regular append only database.
That said, the provided service may very well be a better way to access existing web3 services.
> FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me.
Have you found any uses for blockchain that are not better served, or cannot be served, by traditional solutions?
You did mention Kafka. They got the speed ups exactly how you get views from Kafka: dump them into a PostgreSQL, have an ElasticSearch on top.
The entire throughput of the most popular chains is ~20-50 transactions per second. You can probably handle this immense load at scale that "no traditional system is suited for" on a free tier of any cloud hosting provider.
See also, Web3 Is Going Great: https://web3isgoinggreat.com
There are no mainstream applications and there never will be any.
Can't stand the web3 branding personally.