What Happened to the Semantic Layer? | Hacker News Reader

What Happened to the Semantic Layer?

carlineng.com

3 pointscarlineng6 comments

Is this usable for writing apps too? Why would Malloy be limited to BI?

This looks like it has some nicely well structured patterns for authoring queries. Most places I go, people try Knex.js query-builder for a bit but bottom out fast & start writing raw sql. I've seen a number of homegrown generic query builders, each of which has been impossibly awful to understand & work with.

This looks like a decent take. It's weird to me that it's presented as having such a targeted audience for itself. I'm not sure if there's real constraints that makes it only applicable as such.

Glad you asked! It is certainly usable for writing apps. We publish an npm package, and the Malloy VSCode extension [0] is one such example of an app built on top of it. There's also a demo of a toy CLI app that showcases the simplest possible use of the SDK: https://github.com/malloydata/malloy-demo-bq-cli

Fair critique on the presentation. That's mostly a function of the circles I run in, which is heavily weighted towards data analysts, data engineers and "analytics engineers". But to your point, I think that cohort is unlikely to be early adopters of a tool like Malloy, and that developers are a much easier sell.

[0]: https://github.com/malloydata/malloy-vscode-extension

The cli demo reads very nicely! Most impressive, looks very straightforward. I'd have some questions or playing around to do to understand how best to use this from a we server, where we were making multiple concurrent queries... Should runtime be pooled or are they safe to use across multiple I flight requests? Details. Overall very impressed with the code. Definitely a huge spike of faith for me & greatly accelerates my interest in trying to bring this in as a dev.

We can't be everything to everyone, not a blocker, but I do want to just raise the topic & see what comes out, Malloy seems purely to be a querying tool. So we'd probably have parallel Knex.js code to write data, then Malloy to read it? I should check your road map (yay you have a road map!) to see if there's anything far off here, but it was one of the major other thoughts I had about trying to get onto Malloy.

Hats off on the GitHub vscode notebook. Really really nice malleable tactile way to show the project off. It seems like there is a DuckDB (itself very promising new tech) instance that it relies on, which I think somehow the Malloy vscode extension provided & loads the data into automatically... Is this running on my browser, or GitHub? How? What's the magic that makes this happen? It was really cool seeing the markdown notebook be so tactile, a great feat; really well done!

Thanks for the kind words! Admittedly, a lot of the stuff you're asking about is still a work in progress, and we don't have good answers for it all just yet. The upside of that is you have the potential to influence which direction we take next. If you're thinking about building on top of Malloy, join our community Slack channel, and we'd be happy to provide guidance or take your suggestions/feedback! https://join.slack.com/t/malloy-community/shared_invite/zt-1...

Semantic layers, also known as metrics layers or “headless BI”, have become a popular topic in the online data community in recent years. For all the hype, the idea hasn't seen much traction. In this blog post, I hypothesize why not, and describe the Malloy language and why I think it has a better chance at succeeding.

It looks promising, but I think it's a one sided solution. I would like to see something like Malloy crossed with CubeJS. We need a consistent and flexible language like Malloy and the interface that CubeJS provides: REST, SQL, GraphQL. The way I see it is business defines the processes. Those processes store and manipulate data in a storage. Business processes could change quickly so we need a way to quickly model and adapt to match. The analyst needs a way to slide and dice the data to optimize the business. The end user can get limited access to the data via an NLP interface, such as chatbot. Also, applications and RPA need to interface with the data as well. To make semantic layers work, I think we need to think holistically, not just making it easy for a group of users. It has to be for analysts, developers, and applications/APIs.

Reply on news.ycombinator.com