ksbeking

TL;DR: Sqlfluff rewritten in Rust, about 10x speed improvement and portable

At[Quary](https://www.quary.dev/), we're big fans of [SQLFluff](https://sqlfluff.com/)! It's the most comprehensive formatter/linter about! It outputs great-looking code and has great checks for writing high-quality SQL.

That said, it can often be slow, and in some CI pipelines, we've seen it be the slowest step. To help us and our customers, we decided to rewrite it in Rust to get faster performance and portability so that we could run it anywhere.

Sqruff currently supports the following dialects: ANSI, BigQuery, and Postgres, and we are working on the next Snowflake and Clickhouse.

In terms of performance, we tend to see about 10x speed improvement for a single file when run in the sqruff repo:

    time sqruff lint crates/lib/test/fixtures/dialects/ansi/drop_index_if_exists.sql
    0.01s user 0.01s system 42% cpu 0.041 total
            
    time sqlfluff lint crates/lib/test/fixtures/dialects/ansi/drop_index_if_exists.sql
    0.23s user 0.06s system 74% cpu 0.398 total

And for a whole list of files, we see about 9x improvement depending on what you measure:

    time sqruff lint crates/lib/test/fixtures/dialects/ansi    
    4.23s user 1.53s system 735% cpu 0.784 total
        
    time sqlfluff lint crates/lib/test/fixtures/dialects/ansi
    5.44s user 0.43s system 93% cpu 6.312 total

Both tests were run on an M1 Mac.

Appreciate the feedback! We'll keep this in mind.

There is nothing to host/provision, so it's simple in that sense. You just run it locally with your credentials and connect directly to your database.

It is definitely not the easiest to set up especially when thinking as a team so we'll keep that in mind.

Superset is a beautiful tool focused on self-serve with amazing visualizations. I won't take anything away from them!

Our thesis is that self-serve is much less important than people think, and we find people often make a mess of never-ending dashboards. Current BI tools struggle to prevent that. We solve this problem with a core of software engineering practices.

Yep, I meant protection in terms of intellectual property.

While that's somewhat true, our CLI can push transformations back to your warehouse. We and some of our customer use Quary for their "data warehouse purposes" also. We think the integrated flow makes the E2E experience very quick.

Ben here from Quary.

We love Grafana! It's fab for building dashboards, but it's focused on dashboarding/alerts and on pulling from various data sources, not just SQL.

Quary is purely focused on SQL, and crucially, it allows you to build up and develop more complex transformations.

thanks!

Hey, Ben here from Quary; very valid comments like the one below copied meant we rethought our strategy it a little. We want to be open source but think we need a little protection.

"Hate to derail the conversation, but is Quary something I could easily whitelabel to embed BI into my product for my customers? (Passively) looking for solutions in that that don't feel dumbed down."