Show HN: 0xFast – Faster Web3 APIs
0xfast.com
Excited to showcase 0xFast to HN!
Built using a new indexing system designed for Web3 data, 0xFast outperforms the most popular web3 API platforms, while also being 3x cheaper.
0xfast.com
Built using a new indexing system designed for Web3 data, 0xFast outperforms the most popular web3 API platforms, while also being 3x cheaper.
Edit: I feel like we need somebody like Coffeezilla to unwind the web3 mystery in a digestible format for the rest of the tech world.
1. Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees.
2. This service is better than other web3 api providers.
FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me.
If it helps, you can remove web3 entirely from the problem to understand it. The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have. The other main thing is that there is a standard API that people provide (read: sell) access to, which can be slow and/or expensive if you've got to pull a lot of data from it.
So, Apache Kafka.
Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it
> so you can build something neater because you've got guarantees that many other problems don't have.
So, first you have to find problems for your solution
For services that are using the "append-only log as your source of data" then you need a tool that manages the "expensive to reconstruct it" side of things. I think that is what this does: it materializes and abstracts interacting with the reconstructed view of the data.
Does that give access to the full history quickly?
> Edit: as anyone who actually does this for a living: you never use the append-only log as your source of data. Precisely because it's very expensive to reconstruct it
Yeah I don't want to do that, but I do need to in order to process the data. For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
It doesn't because no append-only log does. Apache Kafka is that "unique web3-only thing never seen before in traditional systems" that no only has been seen in traditional systems, but has been successfully used in those systems at scale dwarfing all of crypto, and has well-understood behaviours and patterns.
Edit: In 2019 Linkedin (who developed Kafka) were processing 7 trillion messages a day: https://engineering.linkedin.com/blog/2019/apache-kafka-tril...
Edit: In 2018 Pinterest was handling 15 million messages a day with 1.2 petabytes of data per day https://medium.com/pinterest-engineering/how-pinterest-runs-...
A view of the final state of data is literally one of the things they require, and everyone knows how to make one.
> For those that don't want to do this, and want to quickly access just the latest state, that's exactly what services like this offer.
So this service just creates a "traditional" database with a "traditional" access method (actually an ancient access method, RPC) with a "traditional" ElasticSearch on top and pretends it's something new, never made at scale, and something no "traditional system" was ever prepared for.
Edit: see LinkedIn and Pinterest above. Or Facebook. Or Amazon. Or AliExpress. Or... "Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie that I can't imagine people who can look at it and say "true, true". The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
Then it doesn't solve the problem people in the space have. I understand the visceral hate for web3, I do, but it really gets in the way of talking about a very bland technical issue.
I think you're misunderstanding what this service is. It's about access to Blockchain data not about running a chain.
> A view of the final state of data is literally one of the things they require, and everyone knows how to make one.
That's one thing it needs, but only one thing. Keeping up to date with this is a problem of monitoring mostly to keep on the right side of any forks. It's not a data problem, but that's not what the API is providing.
> Omg they traditional web2 tech cannot work at the amazing scale of web3" is such a bold-faced lie
It totally would be, but it's not what the site said. It's literally just saying that they're doing better than off the shelf tools because of guarantees most other problems don't have. And I'm not surprised, the data isn't of a crazy scale though is a bit annoying, access patterns are extremely predictable and depending on which APIs you're supporting building a specific thing for it is fairly straightforward.
> The amazing scale of web3 can barely process a few hundred thousand transactions per day, all of them with minuscule payloads.
Nobody is talking about ingest speed really. Unless you're also applying all the state transitions because that's quite annoying.
=== start quote ===
Web2 Databases were meant to deal with datasets that change frequently. They have to provide ACID guarantees to ensure correctness.
The solution to this problem becomes more complex with scale. Distributed transactions, sharding, replication, and fault tolerance are notoriously slow.
Web2 databases are just NOT designed for web3
A system designed to take full advantage of the fact that blockchain data is immutable, and correctness has been dealt with already.
We built a cheaper and simpler storage and indexing system from scratch that can scale to the heaviest blockchain workloads out there.
=== end quote ===
Then here's what you claimed
=== start quote ===
Web3 data is different from what most classic databases are optimised for, because of how it's accessed and the (mostly) immutable guarantees.
The only thing relevant here is that blockchain data is almost exclusively append-only and so you can build something neater because you've got guarantees that many other problems don't have.
=== end quote ===
So, to give you a doze of reality and dispel the lies on the web site I showed that your view of the world does not reflect what the world is:
- Blockchain isn't the only thing to have an immutable log. Apache Kafka is one, and is being used at scale that dwarfs anything in crypto space
- "The heaviest blockchain workloads out there" as claimed by the web site have literally nothing on "traditional/classical" whatever because traditional/classical handle workloads that exceed blockchains by multiple orders of magnitude
- There's nothing revolutionary in the service despite its bullshit claims: they dump blockchain data into a traditional database, slap a traditional (I'd say conservative) JSON-RPC API on top, and probably add a traditional Elastic Search on the side. Everything that they accuse "traditional systems" incapable of doing and incapable of handling. Even though, again, traditional systems can probably handle all this on a free tier of any cloud hosting provider
We've known how to handle append-only mostly-immutable data at scale using "traditional" methods for ages. Literally every banking system is all but required to have an append-only log of transactions for audit purposes. And yet here we are with clueless gullible fools pretending this is something new and something that can't be handled by "outdated slow technologies of yore".
They don't say that standard databases can't handle this. They say that they can do it cheaper and faster without them, because you don't need a bunch of the features. They are literally telling you that their solution is simpler not revolutionary.
You are conflating two very different things, the production and indexing - and then consumers.
> - Blockchain isn't the only thing to have an immutable log. Apache Kafka is one, and is being used at scale that dwarfs anything in crypto space
Sure, and doesn't solve the problem here as you have already told me. Kafka would be ridiculous overkill IMO for the part it would play. The problem is simpler than kafka is designed for, so it makes sense that someone could make a simpler tool to handle it.
> Even though, again, traditional systems can probably handle all this on a free tier of any cloud hosting provider
That very much depends on your users - it's this kind of statement that makes me think you're not picturing what this actually is.
> slap a traditional (I'd say conservative) JSON-RPC API on top,
Yes, this exact API is very much a requirement for this kind of service.
As a final thought, if you think you can throw this together for a large customer base easily and host it basically for free while having better performance - go for it. You'll make money because the alternative options are expensive.
But since this was your last reply, adieu.
"Web3" doesn't lend anything to technical conversations and is best used on board rooms full of coked up investors interested in dropping money on another dumb crypto bro's vision to change the world.
Even so, in a technical setting I often hear engineers refers to "hosted function" or "function service" when referring to "serverless" infrastructure.
Nah. Blockchains are (of course?) not the first database use case where immutability is necessary. Blockchains aren't unique in terms of access patterns, either.
Append only databases have been around for decades.
Heck Kafka is an obscenely popular example today.
I've talked to people working in fields with legally mandated immutable databases, again decades old technology. In the most extreme cases the immutability is enforced using technology, such as a storage medium that only supports writing and not deleting or changing data once it has been written.
Distributed append only databases based on not trusting the people you are doing business with, and where writes cost small fortune, sure, that is new.
But the # times I want to make large transactions where:
1. I don't trust the other party 2. I can't use one of the insanely well trusted escrow agencies, some of whom have been around for literally hundreds of years and have built up trust accordingly 3. I am OK with the transaction not being reversible through the typical legal system remedies
is miniscule.
The idea of a distributed ownership ledger is more interesting, but again, if you have any level of trust with your partners, it is cheaper to throw away the entire proof of X system and replace it with a regular append only database.
That said, the provided service may very well be a better way to access existing web3 services.
> FWIW I'm consulting in this space currently, specifically dealing with blockchain data itself and it makes perfect sense to me.
Have you found any uses for blockchain that are not better served, or cannot be served, by traditional solutions?
You did mention Kafka. They got the speed ups exactly how you get views from Kafka: dump them into a PostgreSQL, have an ElasticSearch on top.
The entire throughput of the most popular chains is ~20-50 transactions per second. You can probably handle this immense load at scale that "no traditional system is suited for" on a free tier of any cloud hosting provider.
There are no mainstream applications and there never will be any.
Can't stand the web3 branding personally.
Seriously web3 is just another way to create hype for arbitrary crypto tokens and Ethereum. In a few years there will be something else - quantum web or something to suck new buyers in.
I have very hard time imagining anyone working in web3 with good intentions.
See also, Web3 Is Going Great: https://web3isgoinggreat.com
- Show HN: Outserv – GraphQL Search Engine [for the blockchain] (1 Aug), https://news.ycombinator.com/item?id=32302368 (11 comments)
- Show HN: I Built 0xFast Stream – 100x Faster Ethereum Block Downloads (22 Sep), https://news.ycombinator.com/item?id=32940451 (8 comments)
- Show HN: 0xFast Indexed Streams – Search and Stream Blockchain Data (16 Oct), https://news.ycombinator.com/item?id=33220456 (0 comments)
Good luck regardless :)
And yet, suprisingly (not) they power everything at scale unimaginable to anything happening in crypto. The "amazing scalable crypto" barely manages the complexity and scale you can easily run from a Raspberry Pi with a sqlite database.
> We built a cheaper and simpler storage and indexing system from scratch that can scale to the heaviest blockchain workloads out there
Translation: you are dumping blockchain data into traditional databases running on traditional servers and backed up by a traditional Elastic Search.
It's ironic that the solution to blockchain problems is so often re-centralization.
It's absurd to claim that something is decentralized when there is always team at the core of this so-called decentralized network whose self-appointed job is to maintain its state of decentralization.
I would argue that there is no such thing as any intentionally decentralized anything, and all claims as such should be treated with extreme skepticism. "trust us, it's trustless."
The only decentralized phenomena are ones that come to us from the universe itself. And even those can ultimately be traced to an origin so the decentralization is only a local phenomena anyway.
Scam. Lies and scam. That is the term for it.
> one of a bunch of different providers all offering the same API to the same public data
It's not the API I object to.
[0] https://www.cnbc.com/2022/03/23/ukrainian-flees-to-poland-wi...
https://www.straitstimes.com/world/europe/cash-crops-dutch-u...
But the pricing seems bizzare.
Hetzner cloud costs 1€ per extra TB (20 included) while this costs 2$ per Gb.
That is an order of magnitude so what am I paying for?
Having dumped several large blockchains into Scylla with indexing in Elastic, I can't imagine a situation where there's enough value-add to justify building a custom database to store it.
Something is definitely wrong with the "Queries per second", it doesn't look like something significant. My usual load for a single node is about 300 req/sec. That's just an average number. Sometimes I do up to 10K req/seq. That consumes some noticeable amount of CPU, but nothing special and the bottleneck seems to be not here.
And I think the main issue is the "Latency" chart. Because my average latency is about 3.5ms. It's like 10x faster. They cannot be so slow.
So I guess the real problem is that the author measures the network. Not the service. So a link to Alchemy happens to be slower than a link to his own server. And the network is the main bottleneck in both cases, because you can easily do 10x of those numbers with a good network.
I wonder further if there is an intermediate solution in this space where chainstate was denormalized into sqlite. How much of the read-only JSON RPC could be implemented with a sqlite over http approach?
Would also be curious to read more about the technical details here. What the application architecture is, for example.
However it seems that there are minor incompatibilities between 0xfast and generic RPC so it fails with ethers.js one getTransactionReceipt. I will look into it in the coming days and get back to you.
"We build and run our own servers. In datacenters."
Is this a selling point today? Suddenly their background in on-prem architecture, operations, security, and network management becomes very important.