HNHacker News
TopNewBestAskShowJobs

bitsondatadev

73 karma · joined August 31, 2020

U.S. Marine turned software engineer turned developer advocate.

https://linktr.ee/bitsondatadev

Disclaimer: I am head of Developer Relations at Tabular and contributor to Apache Iceberg and Trino.

submissionscomments
bitsondatadev··on Why hasn't Presto become industry standard?
Yeah this is a common misconception. Trino and Presto were aimed to replace and speed up the Hive engine.

As you say gwittel, adding Trino to an RDBMS itself won't speed things up. However, if you have operational data sitting in that RDBMS and data sitting in a data lake somewhere on like S3, then you can quickly join those datasets together.

Trino does its best to take advantage of any existing indexes that the RDBMS has by doing a pushdown but won't return that data any faster than the underlying database could. But it's the joining with other data sources data sets that makes the RDBMS connector worthwhile.

If you have a 1GB customer dataset in mysql and a 100TB dataset in s3 of all your orders, then Trino will first run a quick query against your mysql database, get a list of customer ids that meet the query, and then will use that list to filter the order id.

SELECT * FROM mysql.db_name.customer AS c JOIN s3.db_name.orders AS o ON c.id = o.customer_id WHERE c.credit_card_num = 123456789;

bitsondatadev··on Why hasn't Presto become industry standard?
Hey I'm a contributor to prestosql (the one that renamed to Trino). I'll provide a few of my opinions into some of these from the vantage point of our project.

* It's definitely confusing but pretty common in open source projects to see the original creators split off when corporate oversight interferes with the OS governance model. (https://www.computerworld.com/article/2746627/hudson-devs-vo...). This is especially true when, as the OP mentioned, it's a pretty cool tech and a lot of interest in it. Now that the names are different, it is clearing up a bit. We're hoping in a few years there will be one project standing so that you won't have to choose. I don't have to tell you which one I think it is.

* Active-active HA is not really necessary IMO as Trino is designed for low latency interactive queries in general. It can handle longer running batch queries but it gives up fault tolerance to fail fast and you just resubmit the query vs predecessors like Hive, Spark, etc... that handle ETL and long running batch processes efficiently but this adds complexity to the query to checkpoint the work. I could see the need for an active-passive HA to have on deck during a failure. Setting up your own active-passive HA is as simple as putting two coordinators behind a proxy and pointing your workers to the proxy address. Then you basically have the proxy run health checks and flip over in the event of an outage. Here's the issue to track native HA though https://github.com/trinodb/trino/issues/391.

* I'm not sure why autoscaling is said to be difficult. I think this is why you have kubernetes and docker to manage this type of workload.

* The only reason this is a pain to me is that engineers wanting to join our community and commit have a bit of a learning curve and depends heavily on us mentoring and guiding them on how the REST API works, which we don't mind. However, I agree with this choice from a design perspective for the user. If you want to use Trino, it's better not to be exposed to this implementation detail or mess with how this works. It will likely cause you more pain.

* This has improved in the last two years since we branched from PrestoDB 2019 (https://trino.io/blog/2020/01/01/2019-summary.html) and 2020 (https://trino.io/blog/2021/01/08/2020-review.html).

* Agreed, we are working on what's the better model here: https://github.com/trinodb/trino/discussions/6573

bitsondatadev··on Minimally Invasive (and More Accurate) Analytics: GoAccess and Athena/SQL
Thanks! If you haven't, feel free to join our slack if you have any further questions. https://trino.io/slack.html

We're trying to improve the docs and blog about confusing topics. We also started a twitchcast to dig into various technical topics around Trino: https://www.twitch.tv/trinodb. You can catch old episodes here: https://trino.io/broadcast/episodes.html.

bitsondatadev··on Minimally Invasive (and More Accurate) Analytics: GoAccess and Athena/SQL
One quick detail on the Trino description is that not only are some of the initial developers but all of the creators and the majority of contributors (https://github.com/prestodb/presto/graphs/contributors?from=...) and still have contributed the majority of the code in both Presto (https://github.com/prestodb/presto/graphs/contributors)/Trin... (https://github.com/trinodb/trino/graphs/contributors).

To really jump into this, take a look at https://trino.io/blog/2020/12/27/announcing-trino.html.

A few more stats and info:

Trino commits: 22,383 Presto commits: 18,582

Trino slack members: 3,603 Presto slack members: 1,575

Trino supports iceberg: https://trino.io/docs/current/connector/iceberg.html HDP3 Support: https://github.com/trinodb/trino/issues/1218

Trino has addressed a critical security vulnerability that still exists in Presto: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1508...

Give our repo a star if you have a sec: https://github.com/trinodb/trino/blob/master/.github/star.pn...

← PreviousPage 2 of 2