mrtimo

how would you use this within open-web-ui locally?

This is a great lineup. Excited for the "SQL replacement" talks. Malloy is currently my favorite because of it's semantic nature. Define all the joins, measures, dimensions up in the source, and use them as needed. Makes for easy writing, and cleaner thinking.

Also, Malloy has done something clever. They embed duckdb into their VSCode extension, so you can easily query (and join) .csv or .parquet files on the fly. I laugh when I think of all the time I spent noodling with pandas code to do simple data cleaning, analysis, or joining. This also means that you can press "." on any github repo and start querying.

I teach MS Business Analytics students - they really enjoyed walking through an analysis of first names given in the USA for the past century. To see it, click to the github repo [1], press ".", install the Malloy extension, click back to the names1.malloynb file and run the queries.

[1]: https://github.com/malloydata/malloy-samples/blob/main/names...

Malloy is a newish data language with semantic models... Perhaps what semantic SQL would look like: https://www.malloydata.dev/

Enjoyed his roundup in the "Shoving Ducks into Everything" section.

DuckDB is a great tool. In April 2020, the creator of DuckDB gave a talk at CMU. In the beginning he makes a convincing argument (in 5 minutes) why data scientists don't use RDBMS and how this was the genesis of DuckDB. Here is a video that starts 3 minutes into the talk (where is argument starts): https://youtu.be/PFUZlNQIndo?si=ql9n2QuBlAEuGIqo&t=204

In the aerial view image (the 3rd image on the page) you can see the walled city next to a park in the foreground). The park is still there today (and now the remnants of the walled city are absorbed into the park).

Here is an aerial view that has a similar orientation [1].

The park is a cycling park, which explains the curvy paths.

[1] - https://www.google.com/maps/search/kowloon+walled+city/@22.3...

I love using the Malloy vscode extension to explore .csv and .parquet. Malloy embeds duckdb - and is a fresh take on analytic SQL. https://www.malloydata.dev/

Here is the report - released 16 hours ago - that this news piece comes from: https://www.canada.ca/en/health-canada/services/publications...

Just downloaded all the data as a 2.45GB csv file. It took about 10 minutes to "export", before the download started.

Thanks for this background. What's the best way to download your data, or the feds data?

I think I would miss the the semantic data modeling of malloy, and Malloy is definitely easier to read and write IMHO, in part because of the semantic data modeling.

I would also miss the nest feature of Malloy, and the way it handles time / dates / timestamps.

Thanks for letting me know about this! I do the same thing... DuckDB loads CSVs very quick and easy. Malloy uses duckDB and can work in VScode or vscode.dev or the vscode built into github. So, you can analyze large .csv files in any of those places.

For example, go here [1]. Press "." (this loads vscode in browser) and you will be analyzing large .csv or .parquet files directly in your browser, after you click on the .malloynb file

Something I made to analyze every form submitted to the Securities and Exchange Commission (SEC) (10-K etc.) since 2014. [2] Go there, press ".", install the malloy extension, and click on the .malloynb file! Do it all within your browser, or download it and use it in VScode.

[1] https://github.dev/malloydata/quickstart/blob/main/README [2] https://github.dev/mrtimo/isa5

Don't miss the link to the full res cross-section:

https://staging.cohostcdn.org/attachment/11357255-b6b9-4a57-...

I see Malloy as an analysis tool like Pandas, PowerBI, Tableau, or Looker. It seems like most people write SQL to get "All the data" and then take it to an analysis tool for further study.

With Malloy you can do the analysis on your data lake directly for 80% of your questions.

> SQL is great at expressing simple needs very well, but in more advanced cases, it can get "mind-twistingly complicated" even for a mathematically included brain.

I have found Malloy[1] to be easier to read and write, even for really complex queries. Malloy complies to SQL. As an example check out [2].

Malloy can query .csv files directly (enabled by duckdb). You can also join .csv files. This makes getting started a breeze for beginners, and is a boon for data scientists. Much easier than Pandas in many cases.

With the visualization capabilities it has built in, Malloy sort-of competes with Tableau and PowerBI.

Most people only write SQL to get data out of databases, for these people, Malloy is an interesting tool to consider.

[1] https://www.malloydata.dev/ [2] https://docs.malloydata.dev/blog/2023-10-26-malloy-bump-char...

Not affiliated with Malloy, I've just played with it for a while and been impressed. It's also MIT licensed.

I've downloaded many csv files that were mal-formatted (extra commas or tabs etc.), or had dates in non-standard formats. Parquet format probably would not have had these issues!

.parquet files are completely underrated, many people still do not know about the format!

.parquet preserves data types (unlike CSV)

They are 10x smaller than CSV. So 600GB instead of 6TB.

They are 50x faster to read than CSV

They are an "open standard" from Apache Foundation

Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it!

Please promote the use of .parquet files! Make .parquet files available for download everywhere .csv is available!

How is this compare with parquet format?

You don't even need a database file if you use Malloy, just a .parquet file or .csv file will work. Think of a .parquet file as a python dataframe saved as a file. DuckDB combined with Malloy are a powerful combo. If you like writing SQL queries for fun, you should like writing Malloy queries even more (at least I do). If you are looking to compare timeframes, Malloy does this well: https://docs.malloydata.dev/documentation/patterns/yoy

I wanted my students to be able to run open source models. Easiest way was using ollama, and ollama web ui (now open web ui) on google cloud for .18 cents an hour with a 4-core nvidia and 16 GB of ram on a spot instance. I created a tutorial for my students: https://docs.google.com/document/d/1OpZl4P3d0WKH9XtErUZib5_2...

would love to learn more about what you are doing with supermarket prices

It would be really cool if you guys supported Malloy. Maybe you already do? https://www.malloydata.dev/

I am business prof. I wanted my students to try out ollama (with web-ui), so I built some directions for doing so on google cloud [1]. If you use a spot instance you can run it for 18 cents an hour.

[1] https://docs.google.com/document/d/1OpZl4P3d0WKH9XtErUZib5_2...

this is amazing -- thanks. side note: when you want to get more data you can just get the new data, and save it as hacker-news-part-2.parquet - then when you use duckdb / malloy you can read them as a glob (hacker-news*.parquet) - and it is if you were reading one file.

I thought you were joking about the 38 million requests! wow! Have you heard about .parquet files? Save it as a parquet and share the link.

where can i download the data behind this?

I teach an MBA class on a similar subject.

The students like a book called "Adventures of an IT leader". It's a fun read that students with work experience can relate to. And students without work experience can learn a lot from.

This is great. I'm a college prof, and this is perfect for teaching students how to use APIs.

Net result of this action for most people: When traveling internationally, I would use this feature to call family in the US - for free (it only uses data). Now I can't do this anymore, as it will default to using cell networks (not data).

>New telecommunications regulations are being introduced in the EU and U.S. beginning in 2021. To comply with these new regulations, we need to remove the call phones feature in Hangouts. [1]

It appears that the change is due to new regulations?

[1]: https://blog.google/products/workspace/latest-google-hangout...

This is awesome! Plans to include data science topics? Seems awesome for web (and the python course).

Just tried it out for a couple hours, a little feedback on the UI: -After I changed code around, I clicked "play" on the video (mistakenly thinking it would run my code)- of course it resumed the video, and all my code changes/additions were gone. Frustrating for a few times, until I learned my lesson. -After a video finished, I started editing code, but the auto-advance kicked in, moving me to the next video, and ending my code edit.

"Adventures of an IT Leader" from Harvard Business School Press - is good. It is perhaps more general than what you are looking for. It is written from the CIO perspective. You can often find the audiobook in your local library.