tobilg

Nice tool! I built https://sql-workbench.com/ which runs completely in-browser via DuckDB WASM, and enables querying of remote CSV, JSON, Parquet and Arrow data sources, as well as uploaded local files. Charts are supported as well, see the accompanying blog post https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...

I agree to the article, but the discussion is nearly two decades old already and more or less the same since I started in classic BI in the early 2000s.

I also strongly believe in SQL to be the "glue", or the least common denominator for accessing data, that's why I built https://sql-workbench.com which is a free SQL environment in your browser for querying and visualizing local and remote Parquet, CSV, JSON and other data via DuckDB WASM.

It also supports bringing your local LLM for Text-to-SQL generation via Ollama...

Having a look at the docker-compose.yml, I don't really understand why you'd need Mongo, Postgres, Redis and Clickhouse in the same stack... Could you elaborate?

You can do the exact same thing with DuckDB as well...

I'm currently rewriting https://github.com/ownstats/ownstats to this model, with a slight difference that I stream Arrow data from a AWS Lambda Function URL into DuckDB WASM in the frontend... Works great.

An improvement could be having pre-calculated DuckDB database files that are directly attached from the DuckDB WASM frontend, see https://duckdb.org/docs/guides/network_cloud_storage/duckdb_...

You can try DuckDB in your browser via DuckDB WASM with https://sql-workbench.com

I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture could be much more simplified, e.g. running DuckDB on a decent EC2 instance.

In my experience, doing this will yield the query results faster than some other systems even starting the query execution (yes, I'm looking at you Athena)...

I even think that a lot of queries can be run from a browser nowadays, that's why I created https://sql-workbench.com/ with the help of DuckDB WASM (https://github.com/duckdb/duckdb-wasm) and perspective.js (https://github.com/finos/perspective).

Yes, it uses OpenStreetMap data, which you then host on your own AWS account

Right, the diagram is generated from the IaC of the project itself, so it contains all AWS resources involved.

You could use the Lambda function to verify Access Tokens etc. before returning the tile data if that is a concern.

CloudFront as CDN will enable Edge caching, meaning that recurrent requests will serve much faster, as S3 is always region-based.

Exactly! I ran this on a r7gd.4xlarge EC2 instance, which took below 3 hours. Then used the much better upload speed from EC2 to S3 as you described.

Let me know if you‘re interested in access to the IaC repo of this.

That’s not entirely correct. Yes, this project wraps Protomaps/PMTiles and describes the process and delivers Infrastructure as Code to deploy the whole stack to AWS, which the Protomaps website and code repos don’t deliver.

Not sure what you mean with borrowed from the docs, as I have a hard time to imagine a way of delivering the current project without using or referencing specific things from the libraries/projects used.

It uses PMTiles… Range requests are only supported on S3 directly. If you want to use a custom domain, edge level caching etc. you‘ll need to use a CDN like CloudFront

Author here, thanks for posting! The blog post explains how https://sql-workbench.com can be used for in-browser data engineering and analytics. Let me know if you have questions!

The new feature of https://sql-workbench.com uses DuckDB WASM, Langchain & Ollama as well as the DuckDB-NSQL Text-to-SQL model running on the user's local machine to enable querying local and remote data via natural language.

Demo video: https://www.youtube.com/watch?v=rTuCec_fhlk

Thanks you, https://sql-workbench.com author here :-) Let me know if you have some feature requests / bug reports etc. at https://github.com/tobilg/sql-workbench/issues Thanks!

Perspective.js is what I use for https://sql-workbench.com grid and charting functionality.

You can just do the data manipulations in DuckDB WASM and pipe the Arrow data to Perspective...

Thanks and congrats on the launch!

Thank you! It’s really great that the in-browser data analysis ecosystem is really firing up… Exited to see what others build

Have a look at https://sql-workbench.com, it supports query sharing via URL, as well as sharing visualizations. Let me know if you have questions!

The DuckDB WASM space is really heating up! I released https://sql-workbench.com a few weeks ago, which can be used to query and visualize Parquet, CSV, JSON and Arrow data.

There’s also a accompanying tutorial blog post at https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...

Nice! I built https://sql-workbench.com, which is based on DuckDB WASM as well, and can also query Google Sheet directly from the browser.

Here's an example for querying a Google Sheet:

https://sql-workbench.com/#queries=v0,SELECT-*-FROM-read_csv...

Cool, my approach is basically the same, put the schema in the system prompt automatically, and the user prompt from the UI, and render the resulting SQL back to the UI.

Question is more where to host this "on the cheap" because this is a free service, and I can't just spend hundreds of Dollars/month to keep it running... Do you have any recommendations?

Thanks, I‘ll look into that!

Thanks Nico, a small world :-)

I‘ll look into this as fallback method. I don’t like Monaco Editor‘s handling of selections and cursors, it’s quite complicated, but this should be possible to implement…

Thanks for the feedback! Regarding the run query shortcut, if you open the page, there‘s a comment banner on the top of the editor that mentions it. It‘s not the first time that I hear/read that people overlooked it though.

What would be a more intuitive way from your POV?

I‘ll look into the horizontal scrolling/CSV header issues, thanks!

I love Evidence.dev, it’s a great tool!

Yes, unfortunately if the "foreign" sources don't support CORS, you'd have to use a CORS proxy... If you want to self-host, there's one at https://github.com/Zibri/cloudflare-cors-anywhere that can be deployed to CloudFlare Workers (the code is a bit messy though).

GitHub supports CORS for raw data for example, that's why I put it in the sample queries.

Sorry, can you please check again?

https://sql-workbench.com/#queries=v0,SELECT-count(*)-FROM-'...

Thanks!

I'm sorry, I think I fixed this. Can you check? Thanks for letting me know!

Eventually at a later point in time! I‘m currently integrating the AI-based generation of queries…