Trustfall: How to Query (Almost) Everything
github.com
github.com
1 - https://steampipe.io/docs/develop/overview 2 - https://wiki.postgresql.org/wiki/Foreign_data_wrappers
I suspect this is only a basic, naive idea compared to the true potential capabilities Trustfall could unlock.
Your use case even seems to be a UNION and not even a JOIN ?
1. Store the knowledge in a highly structured way and interrogate it with a precise and rigorous query language to extract the exact answer you want based on a well defined set of rules
2. Store the knowledge in whatever ad hoc way it’s produced, and then rely on a higher form of intelligence to take an equally ad hoc query, feed it through the entire universe of knowledge with some attention mechanism, and magically return a (statistically significant) response
Both approaches are so satisfying when they work. Of course you also have everything in between and then you have tools like LangChain that start to bring it all together.
But, there is another option. The tool (presumably AI/ML based) that takes the ad-hoc query and turns it into a precise query to the structured-data tool in order to return precise results.
I hope that this will be Googles answer to ChatGPT... i.e. a chat interface to their existing search tool. This should then be able to "explain" the query its actually generated to the tool as a side-effect.
The focus instead has been in a place where it's easier to stand out: deeper query ability over the data providers than any competing solution, with stronger optimization guarantees. The goal is that any query optimization you could implement by hand, you should be able to implement within Trustfall -- while having the benefit of purely declarative type-checked queries, with integrated docs and autocomplete. If a query is too slow, you don't have to rewrite it -- you can just use Trustfall's APIs to tweak how it executes under the hood and speed it up by using caching, batching, prefetch, or an index.
For a real-world demo of that, I wrote a blog post about speeding up cargo-semver-checks (a Rust linter for semantic versioning) by 2272x without changing any query, just by making some indexes available to the execution of the existing queries. This is awesome because it empowers anyone to write queries, knowing that either their query will either already be fast or it can be made fast in the future in a way that is entirely invisible to the query itself. Link: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
What does it matter that some crates are called trustfall adapters and some are not? You still had to optimise the execution of the query manually. I can see how it’s cool you didn’t need to change the text of the query, but people like SQL because the execution engines are smart enough to optimise for you. They will build a little hash table index on the fly if it saves a lot of full table scanning. The expectations re smart optimisation in the market you’re competing in are very high. If you say it was a one line upgrade to the trustfall executor then people will believe you.
The net result is better than what most GraphQL infrastructure gives you. GraphQL doesn’t give you any tools to avoid full table scans, it just tells you “here’s the query someone wants, I will do the recursive queries with as many useless round trips as possible unless you implement all these things from scratch”. At least your API has the concept of an index now. But I think you’re trying to sell it as being as optimisable as SQL while trying to avoid telling users the bad news that it’s going to be them who has to write the optimiser, not you.
Most uses of data don't have anything like SQL / any kind of a query language, let alone an optimizer. No tool I know of other than Trustfall can let one have optimization levers (automatic or human-in-the-loop) where one can optimize access to a REST API, a local JSON file, or a database -- all separately from how you write the query.
With Trustfall, I'm not promising "magical system that will optimize your queries for you without you having to lift a finger" -- at least not for a good long while. But I can promise, and deliver, "you can write queries over any combo of data sources, and if need be optimize them without rewriting them from scratch." This means that you can have product-facing engineers write queries, and infra-facing engineers optimize them, with both sides isolated from the other: product doesn't care if there's a cache or an index or a bulk API endpoint vs item-at-a-time endpoint, and infra has strong guarantees on execution performance and optimizability so they aren't that worried about a product query getting out of hand and wrecking the system. Trustfall buys operational freedom and leverage across your entire data footprint.
You can see this effect in play in cargo-semver-checks. We use lint-writing as an onboarding tool, because anyone can write a query, and we know we can optimize them later if need be. Both Trustfall and the adapters will get better over time, so queries get faster "naturally". We get efficient execution over many different rustdoc JSON formats simultaneously, without version-specific query logic. And while the hashtable indexing optimizations required some manual work that I didn't time exactly, it was limited to ~1-2h tops and made all queries in the repo faster automatically with no query changes. Rolling out the optimization would be operationally very simple: it's trivial to test, and thanks to the Trustfall engine, I wouldn't have to test it with every combination of filters and edge operations -- if the edge fetch logic is correct, the engine guarantees the rest. Put simply, nobody else needed to know that I made the optimization -- the only observable impact to any other dev on the project is that queries run faster now.
My only question about the actual code is whether you can write these indices to do hash lookups across data sources. Can I avoid table scans when joining two data sets from different adapters?
To be honest, that blog post was targeted at cargo-semver-checks users and r/rust readers, to give them a sense of how cargo-semver-checks is designed and why, with a motivating example of speeding up queries while supporting multiple rustdoc versions. It wasn't really meant to be "Trustfall's entrance on the world stage" even though it kind of ended up being that...
I plan to write more blog posts (and code!) about Trustfall's specific capabilities (and things it can't/shouldn't do) in the future, so hopefully those will come up first in people's searches and give folks the right impression.
Re: multiple adapters, yes, that's the plan. I have some prototype code for turning multiple adapters into a single adapter over the union of the datasets + any cross-dataset edges, and it supports the new optimizations API so the same kind of trick should work in the same way. In general, Trustfall is designed to be highly composable like this: you should be able to put Trustfall over any data source, including another instance of Trustfall, and have things keep working reasonably throughout.
The adapter is free to implement whatever join algorithm makes sense, together with any predicate pushdown, caching, batching, prefetching, or any other kind of optimization.
At a high level, Trustfall's job is to (1) perform any optimizations it can do by itself, so you don't have to do them by hand, and (2) empower the adapter implementation to add any other optimizations you wish to have, without getting in the way.
The goal of Trustfall is to be the LLVM of data sources. GraphQL, OpenAPI, JSON (with JSON schema or not), SQL, RDF/SPARQL -- and none of them can natively talk to each other. Sure, you can stick JSON into Postgres, or compile GraphQL to SQL -- I've done both in production and it's always ultimately a poor fit because you're cramming one system into another when it was never originally designed to support that.
Here's an example: tell me the GitHub or Twitter accounts of HN users that have commented on HN stories about OpenAI. The data is available from the HN APIs on Firebase (for item lookup) and Algolia (for search). I know all of us could write a script to do it -- but would we? Or is it too annoying and difficult, and not worth it? That "activation energy" barrier is something I want to eliminate. Here's that same query in the Trustfall Playground, where it took just a minute or two to put together: https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVl...
Trustfall is designed for interoperation from day 1. It separates the queries from the data providers, allowing the infrastructure to evolve and change how it serves queries without any of the queries noticing anything except faster execution. In practice, that means you don't have to rewrite your product to make it run faster -- which makes both the product side and the infra side happier :)
Here's a real-world example of that. The `cargo-semver-checks` Rust semantic versioning linter implements its lints as Trustfall queries over JSON files describing the package API, and I recently was able to speed up its execution by over 2000x without changing a single query -- just by changing how those queries execute under the hood. More details in my blog post here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
AMA, I guess :)
Tl;Dr implementations of the adapter api can provide hashmap lookups and the query engine will use this when possible before doing a full scan - this is transparent to the query author.
A 3000 word essay takes approximately 10 minutes to read. (Assuming subject familiarity) It is frustrating when the main point is at the end and very little is discussed about the shape of the new api besides the fact that it is experimental.
I spent quite awhile on the site trying to understand the features and benefits of the query language. I left unsatisfied by the long commentless query examples and repeated claims of being able to query anything. It'd be nice to have a discussion of the what the language is, how an adapter should be implemented and how the engine will use my adapter. Maybe that is covered in linked video talks but there are no slides to skim to see if thats worth my time.
OP I appreciate your passion and enthusiasm but as a potential user I can't figure out where to start with the project.
I plan on writing more about all the topics you mentioned, and add docs for writing adapters and queries. I was planning on doing a "Show HN" when more of that was in place, but I'm not the OP here -- someone else submitted it :) Obviously all of the below needs to be more discoverable, and is something I'll work on!
Here are the slides from my talk from the HYTRADBOI conference last year, containing some more real-world query use cases: https://docs.google.com/presentation/d/1foUdlEDOQ1WcTadhAxsA...
This is the current stable adapter API: it's a trait that adapters need to implement https://docs.rs/trustfall_core/latest/trustfall_core/interpr...
It's also available in the Python bindings: https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0....
Tl;Dr of the optimizations API is that every call into the adapter gets a reference to a `QueryInfo` struct. It's totally safe to ignore if the adapter isn't interested in optimizing, and otherwise allows the adapter to ask questions like "from this location in the query, will future operations include traversing edge X and then filtering on property Y?" If so, it's possible to use the API to resolve the expected values for that Y property and use them to speed up the lookup, as in the case with the hashtables in cargo-semver-checks.
The optimizations API is still pretty raw and I didn't want to dig into it too much in that post (which is mostly about cargo-semver-checks' architecture, not just Trustfall internals) in case it changed dramatically. If you really want to see the current state of the API, here's a link to the draft PR: https://github.com/obi1kenobi/trustfall/pull/131/files#diff-...
My hope is that it isn't an "either-or" but an "and". I also believe that Calcite is primarily database-oriented, as are most similar systems. Trustfall is designed to be able to delegate queries (or portions thereof) to underlying systems: for example, you could take a portion of a query and compile it directly to SQL instead of executing it via an interpreter -- no need to worry about query planning over a SQL database when the db itself can almost certainly do that better. In principle, Calcite should be possible to be plugged in like that as well.
Same goes for other systems that do federation over data sources, whatever they may be. Whenever there's an awesome database or federation solution out there that covers some data sets, plug it right in! No need to reinvent the wheel -- Trustfall development can then focus elsewhere, and everyone wins. The goal here is to have a universal data system that works great for any data source in any programming language, not a set of small-time turf wars over whether SQL or GraphQL or $NEW_TECH is better.
Thanks, this made me laugh out loud :)
- Trustfall queries can in principle be compiled into SQL and executed on an analytical database. I haven't built this into Trustfall right now, but a prior project I built that used very similar syntax had this capability so I know it can be done without too much trouble.
- Trustfall's built-in query interpreter is lazily evaluated, and has very strict performance guarantees about what gets loaded into memory when and for how long. I've gone to some rather ridiculous lengths on this, and I look forward to writing it up in blog posts in the future.
- Edges in Trustfall's query graph can be parameterized: they can take an arbitrary number of type-checked arguments of a variety of types, so it should be possible to express just about any kind of join, API endpoint, reconciliation function, window function, etc.
- Whether a data source is large or small doesn't matter to Trustfall. So long as the data provider implements the Adapter interface, it can be plugged in. Whether the underlying data is read into memory all at once or progressively is the adapter implementation's business only -- Trustfall is extremely laissez-faire on this.
- Multiple data sources can be combined together, and queries written over all of them. This is something I'm working on right now, and hope to have more to share on in coming months.
It sounds like you might have a specific use case in mind, perhaps? Feel free to DM me on Twitter if you'd rather not discuss publicly.
SQL is also really awkward around APIs, since API norms and schemas don't map cleanly to SQL tables and JOINs. For example, many APIs take arguments, and while the most natural way to express an API call in SQL is as a SELECT or JOIN, requiring user-provided arguments there is really clunky. Worse, if the API that takes an argument sets a default value if that argument is not provided, then you end up in a situation where a `SELECT * FROM API` can return fewer results than a `SELECT * FROM API WHERE <some predicate>`. Using a new language meant that I could design it to best highlight Trustfall's capabilities with minimal work.
Trustfall's core is language-agnostic, so it should be possible to put a new language parser in front of it in the future -- including SQL. So I haven't ruled it out forever, I just didn't want the first thing I built to be "SQL but worse" because it's in a domain poorly suited for it.
[1]: https://twitter.com/sc13ts/status/1413729761247916036
[2]: https://www.scattered-thoughts.net/writing/select-wat-from-s...
In https://www.hytradboi.com/2022/how-to-query-almost-everythin... @ 4:15, you mentioned data modeled as graphs are equivalent to relational models.
I can see relation models can be represented as graphs (e.g., table foreign keys to another table are "edges"; fields in the table that are not primary keys are "properties")
However, I am not quite confident in the claim that arbitrary graphs being mappable as relational models. I suspect there are certain graphs that can be difficult to express as relational models.
So question for you: Would you say all Trustfall expressions can be translated to SQL? Or, there are cases where there's no such translation possible?
But from 7 years of working on graph databases, and even participating in some graph query language standardization work, I know two things:
- Anything a graph database can represent and store, a relational db can also represent and store.
- Trustfall's representational power is broader than what either of those would reasonably represent.
I say reasonably because, for example, with sufficient contortions you could make SQL to give you function-like edges like you'd expect in an API. Table-valued functions are awesome! But if every one of your joins was via a TVF ... you'd have a number of problems.
Examples: in Trustfall, you can write a query that asks for the prime factors of all numbers in a given range, without having precomputed or stored any of that data -- it can be computed and loaded "on-demand". You can also have an edge like `divisors_that_are_multiples(of: Int!)` that requires an integer to be provided before being able to be resolved. Both of these are possible to represent in modern SQL, if you try hard enough. But should you? Absolutely not, or you won't be happy with the outcomes.
To make true datasource-agnostic querying, you can't start with a "preferred datasource" that everything else must translate to. Either all data sources are equal, or else there's a massive incentive to use the preferred one at the expense of all else -- so there are massive incentives to optimize predominantly for the preferred source, and we're right back at square one.
[1]: https://www.hytradboi.com/2022/how-to-query-almost-everythin...
Realistically, the docs are still very sparse -- I wasn't planning on posting this for another few months, but someone else noticed my project and beat me to it!
New data providers need a schema and an Adapter implementation.
For Rust, this is the Adapter trait that data providers need to implement: https://docs.rs/trustfall_core/latest/trustfall_core/interpr...
This is the equivalent abstract class in Python: https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0....
For Python, the package docs have some (underwhelming) additional examples of building a schema and passing an adapter to use to run a query: https://pypi.org/project/trustfall/
The process in Rust is equivalent, here's the code powering the demo in the talk: https://github.com/obi1kenobi/trustfall/blob/main/demo-hytra...
I realize this is all extremely suboptimal, but like I said, I wasn't ready Please stay tuned! I'm planning to spruce all of this up in the coming weeks!
Reading further, it seems that it is actually a variation of GraphQL.
I've never used GraphQL before, maybe one can get used to it, but it doesn't look very nice for defining data queries.
No language syntax makes everyone happy :) But it's definitely is something that isn't hard to get used to -- in my experience, people go from "never seen it before" to "can productively write their own queries" very quickly after trying a few examples. Try out the example queries over HackerNews in the Playground (ideally not on mobile) and see for yourself: https://play.predr.ag/hackernews
And in the long run, Trustfall is more meant to be like an "LLVM for data sets," with pluggable language frontends. There's already been interest in plugging in other languages and making them generate Trustfall representation internally, to make use of the rest of its functionality without using the language itself. This is fine too!
Are you planning on doing reactive/live/materialized queries? You could use a config file to specify the live queries (in the Trustfall DSL) which can be fed to the engine on startup.
In particular, while that tech is really cool, it's not clear to me how many use cases for it are actually ready to adopt it. I'm hesitant to build something that looks really cool, but is somewhat complex and doesn't get used a lot.
For more context, see https://ontop-vkg.org/guide/
edit: Admittedly the website/video tends to talk about data sources a bit hand-wavy. I'd have loved some real world examples on how one goes about adding a data source. Also how we handle problems of scale.. ie passing filters to the data source, rather than drinking from a firehose and filtering after the fact.
With that said, the idea is becoming interesting to me. At the very least i am liking the idea of a standardized query interface to "things". Just feels like edge cases might drown me.
- It could be used as the "ingestion process" itself: write the ingestion as a Trustfall query (or queries) over the data sources, and store the results in a traditional database or other system.
- It could be used to make the "ingestion" a mere implementation detail: you could write a query over all the data sources, and that query could be executed against the raw data sources themselves, or some "ingested" format which would presumably be faster and more efficient. The query can't tell the difference, and this frees up the ingestion system to evolve independently of how it's being used.
It sounds like you might have a specific use case in mind? I'd love to learn more about it if so! My Twitter DMs are open, or I could give you my email address if you'd prefer that.
There's definitely a bit of a learning curve because the docs are few and far between, but in general, everything is much simpler than it seems. The people that have adopted it already have been able to do so within a couple of hours with minimal guidance from me.
For passing filters to the data source, the API there is still very experimental (it was just an idea in my head until last week!) but I've been working on it recently. Here's a blog post showing a small bit of that new API and using it to dramatically optimize a workload: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
This is my full-time project right now, so stay tuned and expect a decently steep progress trajectory :)
Somehow it didn't make it to the mainstream.
The query language is substantially more powerful than GraphQL, much closer to SQL. And the idea is unifying and querying any kind of data, not just websites but also databases, raw files, ML models, anything at all.
For example, the cargo-semver-checks semver linter for Rust uses Trustfall to abstract over the different JSON formats in which different Rust versions describe a library's API. The lints are written as Trustfall queries (GraphQL isn't expressive enough for this!) and don't care about how the underlying data is represented. Recently, I was able to tweak how those JSON files are used to serve data, and sped up cargo-semver-checks by over 2000x -- without changing a single query. More info here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
If you want to see more of it in action, there's a 10min talk featuring demos and real-world use cases here: https://www.hytradboi.com/2022/how-to-query-almost-everythin...
@obi1kenobi, could you comment a bit on the motivation and background to this project?
The motivation is that queries that are trivial in a SQL database are painfully impractical over any other data source. Most query languages are tied way too closely to a data format or representation, and fail miserably at the boundaries to other systems or representations. Yes, we can e.g. plug SQL into APIs but there's a severe impedance mismatch there -- we end up forced into all sorts of unintuitive situations and poor UX because SQL was never meant for APIs.
Based on experiences I had at work (some of them covered in that talk!), it felt like an "LLVM for data sources" could be useful -- that's what Trustfall is. Yes, it includes a query language. That's because SQL and GraphQL and the like just weren't a good fit to demonstrate true datasource-agnostic queries: we want a modern GraphQL-like user experience, with SQL-like expressiveness but without limitations that make it hard to plug in APIs and ML tool. But the query language is just one layer of the system, and it's entirely possible to replace it with another language parser while keeping the rest of the benefits.
A key benefit of Trustfall is that you can write queries over any data sources, and then change the underlying implementation to add optimizations or to migrate to a new system without the queries needing to change. For example, cargo-semver-checks, a Rust semver linter I built on top of Trustfall, supports 4 separate format versions of the underlying data source it uses with the same set of queries, and I was recently able to speed up its execution by over 2000x with a tiny amount of code and effort and without even a single query needing to change -- just by tweaking how the data adapter executed queries to plug in some indexes where they mattered: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
Excited to see what you build, please post a link or email / DM me on Twitter if you'd rather not post publicly? The docs are still sparse (I wasn't planning to do a Show HN for another couple of months, but someone beat me to it today) but I'd be happy to help with anything. There are Python and Rust bindings, and feel free to open issues for anything that's confusing:
There's no measurable MoM growth yet. This is the first major look the broader community has had at the system outside of a relatively small conference where I gave a talk, and demos I've given to friends and colleagues. I wasn't planning on submitting to HN for another few months -- but someone else beat me to it today :)
There are production-grade adapters for HackerNews APIs and for Rust's rustdoc JSON format, used by the Playground (https://play.predr.ag/hackernews) and the Rust semver-checking linter cargo-semver-checks, respectively. There are also a few more demo adapters in the Trustfall repo itself: https://github.com/obi1kenobi/trustfall I expect the number of adapters to grow significantly in the coming months, so please stay tuned and let me know if you have datasets you'd like to try it with!
The "largest dataset" is a bit of a trick question: how big is the dataset of "all HackerNews data available via its Firebase and Algolia APIs"? Because that's what the Playground queries.
The Rust semver linter has been used by dozens if not hundreds of crates, and the JSON payloads in question there are in the 100-400MB range. The example in this blog post runs 40 quite complex Trustfall queries (they express semver rules!) over 400MB across two JSON files in 8 seconds: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
You can also see more real-world use cases in this talk I gave last year: https://www.hytradboi.com/2022/how-to-query-almost-everythin...
Probably around 200-300 people or so have written queries in the language, with experience levels ranging from seasoned engineers to Excel analysts and SQL analysts with no programming experience outside of that domain. The language itself is actually a refinement from an earlier open-source project I developed and open-sourced through my previous job. That open-source project was used to query everything from a multi-TB SQL cluster to APIs to ML models, and with Trustfall I've taken the opportunity to revisit and update design decisions that the previous project got to regret in retrospect.