Obsidian Dataview: Turn Obsidian Vault into a database you can query from
github.com
github.com
I sometimes wonder if learning how to use Dataview inside of Obsidian is too technical to get mass adoption, compared to WYSIWYG tools like Notion that let you build databases with nice UI filtering. But the performance of Dataview is so much better, and the control you have over queries so much more granular, that it’s worth the learning curve. The Obsidian community is also incredibly good at helping new users learn, which makes me feel optimistic.
To help make things a bit more user-friendly I have been building an Obsidian theme called Minimal[1] which allows you to take Dataview tables and display them in a card layout[2]. I really enjoy that layout for certain types of information, while having all the underlying power of Dataview.
I expect these tools to include more and more tools for automation based on a reactive programming style (more suited to non-programmers), expanding its function as a warehouse of ideas into a more active role for user-created applications. The NOCODE camp is growing from its VB6-like visual-style roots into environments that look more like an IDE in the style of the browser's "Inspect" window.
The key development that I miss is that of reifying query results, allowing the user to use the data generated by a search as an object in itself. I even have a name for that reified object - a wit would be a collection of data compiled with a common criteria (representing a minimum unit of meaning, is a wordplay on bit being the minimum unit of knowledge), similar to a spreadsheet range. When tools like Obsidian or Logseq start allowing the user to use query results as first-class data, I predict that we will see a revolution in end-user process automation.
Yes, this https://youtu.be/Pot9GnHFOVU is the kind of programmable system I'm talking about, creating behavior on top of your stored content with methods specific to the particular collections of data already in your system, and ideally with minimal syntax for transferring data among the different tools.
Outliners are used in this area for decades. What we have now is just a new hype-cycle, with new apps, new technologies and a new mix of features.
> When tools like Obsidian or Logseq start allowing the user to use query results as first-class data, I predict that we will see a revolution in end-user process automation.
So, SQL?
To say that what we have now is nothing but a new hype cycle while the Zettelkasten method was only widely exposed within the last couple of decades, and was not utilized in the design of past Outliners, shows you don't really understand what's actually changed.
So what exactly don't I know? Genuinely.
Luhmanns method is simply not that special, nor unique. Many scientists and artist used similar methods and tools. He is mostly just replicating what was common at the time, but with some twist. And in the 80s or 90s is would be really awesome, probably even somewhat groundbreaking. But today? It's just another graph-system which you find in wikis or hypertext for decades now. And most people are not even using the benefits of his number-system, just calling everything zettelkasten which is just a wiki otherwise, leaving even less of his originality. Though, in digital space it make less sense anyway, so not a big lose.
But for whatever reason, there now this hype from people who seem to have know knowledge about those space or it's history, or even really understand the terms, thinking Luhmanns method is called Zettelkasten...
What outliners do you know which allow end-users to feed their data into formulas for processing it without using general-purpose programming languages? I'm genuinely interested, as I've been exploring these tools for years and I haven't found any that meets my requirements.
> So, SQL
How many end-users without any programming experience are able to create moderately complex SQL queries?
With a data-centric outliner tool, they'll be able to decompose those into several simple graphical queries.
Bit of a pointless constraint, the talk is about outliners, not no-code-datamangment. Which tool today does this even offer on a useful level?
But you can look at leo editor (https://leoeditor.com), which is active for 20+ years, fully scriptable and extendable. Though, it's a hot piece of garbage for laymen. It's offers a bunch of features and plugins even for non-coders, but I'm not sure it would satisfy you for this area, if you can't code. Obsidian is probably better than, even if it has more limitations at the moment.
But I'm not sure if there ever is a tool which will satisfy everyone with just a no-code-approach.
The point I'm making is that those are both aspects of the same substrate, and modern knowledge tools are exploring the gap between those two - as well as expanding them in new directions.
Yes, certainly Leo is one of the closest tools on which to build a system like that, although it still relies too much on Python programming (and its usability leaves a lot to be desired, even if the concepts are solid). I expect that the next popular tool in that space will grow either from online notebooks like Jupyter or from desktop linked note-takers like logseq.
(BTW I can code, but I don't want to; remembering the syntax details of the needed APIs does not come natural to me, so it's usually not worth the effort). I still believe that it is possible to grow a programming tool similar to Excel but with structured hierarchical content, without requiring coding more complex than spreadsheet formulas and filters.
Do they? As I see it, they make it worse. We have all kind of specialized languages. Each function coming with it's own limited DSL and API. Often with poor documentation. Is this actually better?
Yes, each language has on it's own less complexity, but all in all they have more problems and limitations than a proper general purpose language with a good API. I don't think that this is really a good future. But it's true that it might lower the bar for new people.
> BTW I can code, but I don't want to; remembering the syntax details of the needed APIs does not come natural to me
So the problem is not the language, but the support? More autocompletion, documentation, automation, etc. That's a concern of mine too. Things are complex, but there is no support for the casual users. Actually I think if obsidian had been build ontop of VS Code and gained instant support from all the fancy developer-tools, it would gain significant benefits.
Another solution could be integrating an interface like Node-RED offers it. This seems to be quite accessible for casual experts.
> I still believe that it is possible to grow a programming tool similar to Excel
And Excel has good access for new users?
That may be because your development needs are well served already if you're a developer, so you won't see the value in such system. For people who rely on others to build every single piece of automation they need, being able to create their own small automatisms would be priceless.
> all in all they have more problems and limitations than a proper general purpose language with a good API
End users don't need a general purpose language with a good API - at least not as someting that they will handle directly. They need small, self-contained domain-specific languages that they can expand in areas where they fall short, but without the full power of a general programming system (which would overcomplicate the development environment and provide too many opportunities to shoot themselves in the foot).
> So the problem is not the language, but the support?
For me yes, partially, because I already know programming. I could work with a programming environment with complex syntax that autocompleted the required syntax to match my thoughts (something like GitHub copilot works in that direction). But that's still two points of complexity above the system I envision as possible.
Something like the classic Hypercard could be a good starting point for the mental model of such tools. A simple data model, integrated storage, and interactive processes using a programming language with minimal syntax. Expand it with hierarchical storage, expanded integration with other systems and a modern declarative language, and you're set.
Someone today pointed me to a system like this[1], Lepiter, which looks a lot like the solution end users could manage to create on their own on top of their knowledge systems: like a browser with separate cells, where each cell can run a different runtime, hyperlinked and with a simple storage mechanism.
[1] https://lepiter.io/feenk/introducing-lepiter--knowledge-mana...
> More autocompletion, documentation, automation, etc.
Those are good things but not enough. What end users are lacking the most is the possibility to create their own automated processes; most tools only allow them to choose among a library of predefined ones, without allowing them to grow the library.
> Another solution could be integrating an interface like Node-RED offers it. This seems to be quite accessible for casual experts.
Unfortunately, graph-based flow visual tools have proven to be useful only for a small subset of automation tasks largely consisting on data transformations using pre-defined functions. The most promising results in End-User Development point out that reactive, rule-based runtime environments like the spreadsheet seem to be better abstractions for allowing users to define their own computations.
> And Excel has good access for new users?
Yes, spreadsheets have a low learning curve and high ceiling. And compared to imperative programming languages, they're like walking up the slope of your house vs. extreme rock climbing.
I don't think it's all that useful, outside of some basic "making a shopping/packing/project list" type calculations. It's worth checking out if you're interested in the history though.
You contradict yourself. Is it "just a new hype-cycle," or is it "new apps, new technologies, and a new mix of features?" If it's the latter... isn't that substantively different than what we had before?
Outliners have quite the fandom, but my issue with the traditional outliner is that they often trap data in outline form. Sure, most good outliners offer export options. Unfortunately, the second you've taken data from your outliner and exported it, you now have two versions of your data, and you've lost.
Logseq (and Obsidian) have changed this. Finally, a single document _is_ an outline, a backlink, a data source, a webpage, a presentation, and so on... The only thing the user has to do is work with that single page of information.
It's more than hype for me.
The concepts are old and have a new hype. The apps are new. But generally, it doesn't matter much for the user whether you have some old-school desktop-app with gtk or Qt, or a "modern" electron app, or some webapp on your own server or some service in the cloud. At the end it's all the same ideas and concepts in different colors and flavors.
> If it's the latter... isn't that substantively different than what we had before?
That entirely depends on who is "we". We which are all users? Or we the old users? Or we the young users? Or we the new users? I'm not saying that there is not development or change, just that this is nothing new, or even a new development. This has going for a long time, it just reached a new cycle and and a new generation of users.
> Outliners have quite the fandom, but my issue with the traditional outliner is that they often trap data in outline form. Sure, most good outliners offer export options. Unfortunately, the second you've taken data from your outliner and exported it, you now have two versions of your data, and you've lost.
This is a fundamental problem of every app and space. But interoperability does exist here and there, it's not just exports.
> Logseq (and Obsidian) have changed this.
Not really? They only established a common foundation for interoperability. Which is also what other apps are doing to some degree. But you still have limitations by everything which is not covered by this foundation. For example, can Logseq handle dataview-syntax? Or generally any obsidian-plugin has it's own special syntax? What about other markdown-editors? How much do they support the special syntax used by obsidian or it's plugins?
True, the markdown-space has now a healthy movement toward interoperability and gains many nice functionalities. But it's not like this is especially new. And all those features also existed before Markdown in some app, in some way or another. Nothing is really new here. What's happening is that a new platform is growing, collection nice stuff from the collective pool of abilities, and a new consolidation is happening. This happens all the time. Sometimes it works, sometimes it dies.
Funny enough, the WWW was one of the first and most successful of this movements. And markdown grew from this as a simplification, to now grew into the same space where WWW started and obsidian now moves toward too, just in a different flavor. This whole "new" movement is just replicating where others were already before in different cloths.
> Finally, a single document _is_ an outline, a backlink, a data source, a webpage, a presentation, and so on...
So has Org-mode 10 years ago, XML 20 Years ago. HTML 30 years ago. More or less, the numbers may be a bit off.
> It's more than hype for me.
Good for you. I'm not question that. Just saying this generation is not the first one having that.
HTML was a step in the right direction (and XML a misstep); before it, you needed to program a GUI API to lay out content intermixed with interactive components. Saying that the web brought nothing new because BBSs existed before it is missing out on the importance of the details in achieving that result. Precisely, "different cloths" is what makes the movement relevant to make it accessible to new users who could never have used the previous system.
You're right in pointing out that interoperability is a key factor in the new batch of tools. The key point that neither Org-mode nor SGML-derived codes had is is to make user-entered content accessible to transformation tools without the need to code the transformation in an imperative language. Spreadsheets had this feature as well, sure, but spreadsheets didn't easily allow for hierarchical content.
The main new aspect that logseq and Obsidian bring to the table is the possibility to backlink to any part of the hierarchical structure. The tools automatically compile all those links as a new data object, which the user can process without creating a script to gather all that data. HTML had the potential to popularize that feature 30 years ago (with a slightly more awkward syntax than Markdown), but browsers never got around to it.
Previous tools would require users to move content to a different tool (say, by copy-pasting the content of your files to a spreadsheet to apply formulas), destroying the possibility of having a central personal knowledge repository. In logseq / Obsidian, you can keep adding multimedia content to your structured knowledge base, and exploit that content without coding at all, or at most by creating simple declarative expressions. What tool before these ones had that possibility before? With hierarchical structures, not just flat structures like the spreadsheet?
No, I know that, I just don't think it's a relevant problem for the user group we are talking about.
> Saying that the web brought nothing new because BBSs existed before
I did not say that? And in the first place, BBS competes with other networks like the Internet, not a service running in such a network.
> The main new aspect that logseq and Obsidian bring to the table is the possibility to backlink to any part of the hierarchical structure.
No, they do not. Linking is a fundamental concept of hypertext. And easy linking inside your dataspace, as also automatically listing backlinks, was already established with wikis 20+ years ago. Even hierarchies exists in wiki space for a long time now in several different ways.
> The tools automatically compile all those links as a new data object, which the user can process without creating a script to gather all that data.
What kind of low-level-tools have you used till now that this is your level of knowledge here??
> In logseq / Obsidian, you can keep adding multimedia content to your structured knowledge base, and exploit that content without coding at all
Yes, because someone else does the coding, as always. This is the benefit of a popular ecosystem. But this is not the point of having a scriptable and programmable app. The scripting just enable you to personalize on small levels. The programmability should enable the user and community to extend the program with new abilities, as dataview here is doing it. There are other extensions who enable scripting to certain degrees for the user, as obsidian is not really adding this on it's own.
No app will ever be perfect for everyone. That's why extensions and scripting are beneficial, especially for this space.k
> What tool before these ones had that possibility before? With hierarchical structures, not just flat structures like the spreadsheet?
The constraints again. Why does it matter that obsidian&co. can do some more tricks than their ancestors, like playing videos, when the fundamental purpose and handling is still the same? That's like the old joke of taking something old and just add a clock, to sell it as new. More bells and whistle don't make a new church.
What user group are you talking about? I'm specifically talking about the large majority of users who could never learn to program in a syntax-heavy general-purpose programming language! X-D Those people nevertheless have a need to build their own userland applications, and no current tool serves them well; but a Nocode programmable knowledge database could do it.
> No, they do not. Linking is a fundamental concept of hypertext. And easy linking inside your dataspace, as also automatically listing backlinks, was already established with wikis 20+ years ago.
And as I explained, no end-user tool has exploited that possibility to allow end users to access those automated links programatically with nocode tools (like for example with spreadsheet-like declarative expressions). The browser missed the chance by having no persistent storage and only allowing javascript as their programming language, and wikis never were nocode-friendly. Yet modern tools are just beginning to do that on top of a personal knowledge base.
> What kind of low-level-tools have you used till now that this is your level of knowledge here??
I beg you pardon? I've programmed all the way down from building my own flip-flops with electric transistors up to non-linear mathematical solvers for combinatorial optimization. I don't see the relevance, since low-level-tools will never be adequate development platforms for end-users.
> Yes, because someone else does the coding, as always.
No. Logseq / Obsidian have tools that allow end users to build their own coding through templates and queries. The kind of coding is different from scripting, though; it's more like creating semi-automated workflows that put all relevant information in front of the user, which then proceeds to transform it with direct manipulation. What tool in the past has allowed end-users to create that kind of workflows from end-user templates and arbitrary queries, other than non-hierarchical spreadsheets?
> The constraints again. Why does it matter that obsidian&co. can do some more tricks than their ancestors
The constraints are there for a reason, and you don't seem to grasp why they are important. The spreadsheet , and spreadsheets are typically not automated through scripting and extensions - using those is normally considered a failure of the spreadsheet model, because you're processing data outside the mental model of the spreadsheet as a programming environment i.e. cheating. Most End-User Development tools rely on the assumption that users can't program at all, so all automation concepts must be presented to them through alternate syntax and interactions, quite different from classic coding.
A spreadsheet-like development environment based on outliners as its data structure, rather than the tables of a spreadsheet, should also be programmable by end users without relying on imperative scripts external to the tool. Logseq is creating the basis for that storage model, which some people are starting to exploit the kind of semi-automated workflows I talked about above.
> like playing videos
That's a red-herring, not a main point. I mentioned multimedia merely because programmable tools tend to be text only, forcing users to use separate tools for rich content, or import specific libraries to handle multimedia data as programmable objects (rather than editing them as native interactive content like they would on the GUI). The point is that end users should be able to treat all kinds of content equally with their tools, not being tied to the Unix shell constraint that all content is moved between processes as text streams or binary blobs.
The kind of people who use such tools. The kind of people who can lookup DSLs to create dataviews, or copy&paste snippets to make up stuff. Something like a knowledge-system is not used by children or someone similar.
> Those people nevertheless have a need to build their own userland applications,
No, they do not. Not every user around the world has this demand. Most user don't have this demand. We are talking here about a minority of quite special usecases. Most people are not even able to mange their own files, and you mean they will use a complex knowledge-managment-system? People are even challenged with a simple todo-list or note-app. No way will they use something more complex.
> and no current tool serves them well; but a Nocode programmable knowledge database could do it.
No, it's not. Nocode is not magic. You still need a certain level of competence, understanding and motivation to make something with it. And whatever interface you present to the users, be it GUI or text or whatever, there always will be a certain level of complexity that not any random user can master. And this is good, we already see with apple and google what harm we have from too much simplification to cater to the uninteressted.
> No. Logseq / Obsidian have tools that allow end users to build their own coding through templates and queries. The kind of coding is different from scripting, though; it's more like creating semi-automated workflows that put all relevant information in front of the user, which then proceeds to transform it with direct manipulation. What tool in the past has allowed end-users to create that kind of workflows from end-user templates and arbitrary queries, other than non-hierarchical spreadsheets?
Many? What? This is also a kind of scripting, and there is an endless amount of tools enabling people to do that style of scripting. Starting with shellscripts, batchscripts, all the strange scripting-engines. Heck, emacs and has this style as their main purpose established. The data science-sector has a very long history of building such tools on a more professional level and integrating data-views. Wikis and databases have a long history of scripting in the knowledge managment-space. The likes of obsidian are just the next iteration, there are not fundamentally new.
> and spreadsheets are typically not automated through scripting and extensions - using those is normally considered a failure of the spreadsheet model
No, they are? What? People have build all kind of crap with VBA for a very long time. There is a reason why this garbage is still around, getting ported to everywhere. And let's not start with the plugins or how extensions for MS Office are called these days.
> using those is normally considered a failure of the spreadsheet model, because you're processing data outside the mental model of the spreadsheet as a programming environment
Maybe that's our bubbles understanding. Seems the rest of the world doesn't give much here. Limiting yourself to a model is nonsens anyway. No model is ever perfect and satisfying every user.
> That's a red-herring, not a main point. I mentioned multimedia merely because programmable tools tend to be text only, forcing users to use separate tools for rich content.
It's not like obsidian is doing more with them, just because it can play them. Until it understand them can make them accessable for the rest of the system it's remains some fancy gadget.
> No, they do not. Not every user around the world has this demand. Most user don't have this demand. We are talking here about a minority of quite special usecases. Most people are not even able to mange their own files, and you mean they will use a complex knowledge-management-system?
There I have to strongly disagree. There's a whole academic field of study, End-User Development, dedicated to prove that idea wrong. Every computer user I know is able to use spreadsheets to create their own data types tailored for their domain-specific use cases, some times quite complex ones. That they can't use scripting languages is a failure of those languages, not of users.
And yes, users are able to handle complex knowledge management systems when they're built from the ground up with the right models, instead of relying on arcane legacy metaphors from the 1960s, such as hierarchical filesystems - which are good for developers, not so much for end users. Our knowledge about digital content organization has evolved a lot since those early pioneer times.
With the proper tools, of course they'll be able to create new software artifacts, which of course they want and need - the same way they want and use new apps built by others. If they could create those themselves, why wouldn't they build their own?
> Many? What? This is also a kind of scripting, and there is an endless amount of tools enabling people to do that style of scripting.
True, but you're missing out the possibility of enabling people to create automation without any scripting at all. There aren't many tools for that other than spreadsheets, which have bad integration with the rest of the computing environment.
>Wikis and databases have a long history of scripting in the knowledge managment-space. The likes of obsidian are just the next iteration, there are not fundamentally new.
I recommend you to learn about End-User Development,[1] to see all the ways these tools can be expanded beyond classic scripting, to see what you're missing out, that other people in this thread are aware and you're not. The end-user computing landscape is going very soon to be transformed, flooded with the kind of tools that end users can build for themselves, for the very same reason that the "VBA and MS Office plugins garbage" are still around; just this time, we finally have a chance to create a way better platform for those tools, which doesn't rely on outdated plugins and low-powered scripting languages.
"Watch what I do"[2] and "Your wish is my command" [3] are the seminal works with loots of good examples. They're a bit dated, but some of their best methods have still not found their way to practical common-use tools (but modern outliners could change that very soon).
[1] https://en.wikipedia.org/wiki/End-user_development
[2] https://archive.org/details/watchwhatido00alle/page/n5/mode/...
I think we're getting there, too. See, for instance, https://github.com/nothingislost/obsidian-query-control — it renders the results of an embedded Obsidian query and you can interact with checkboxes. Likewise, a collaboration between dataview and the Task Collector plugin might let us manipulate different task statuses from a dataview query: https://github.com/ebullient/obsidian-task-collector/issues/...
Exciting times!
Thanks! I had it as a vague idea for several years, though it has been getting a lot more concrete as I've seen new tools adopting similar approaches - but without any of them taking the final step of converting a search result collection into a user-accessible object ready for direct manipulation.
And now that I see tools adopting that style of data management, the concept allows me to see the commonalities among tools with quite different approaches - be it tabular Nocode storage like Airtable or outliners like Roam, Workflowy and the like. Whenever the tool returns a collection from a data gathering process, I see a wit that should be able to be extracted and further processed by the user.
BTW, the main difference between a query result and a wit is that you can store its items in the same structure as the rest of your content in the outliner, and send them to a new process (like spreadsheet cells, but with hierarchical data). Usually, query results are presented through a different interface than the rest of content, but a 'wit' should be homoiconic like Lisp.
(Errata to my grandparent post: a 'bit' is a "minimum unit of data", of course, not of knowledge).
> Exciting times!
You bet! :-D
There is a fair bit left to do for the plugin (export to raw Markdown, section-level metadata), but I hope you find it useful!
It has completely replaced Pocket/Instapaper for me.
That's all pretty abstract. A simple example: it allows you to construct a tabbed view of several todo lists based on tags and fields and adjust priority and status from that single UI.
It's been available for many years, but has never been trendy, despite its unique design and strengths. I recommend giving it a spin!
I was a heavy TiddlyWiki user in the past but moved away from it precisely because of its singe-HTML file design. Modern browsers like chrome made it virtually impossible to save an edited page. Not TW's fault per se, but ruined my experience with it. There is a way to install TW-5 with node.js as a directory tree but my ship has sailed. Now I am a happy obsidian user.
Not to mention downloading and parsing 20mb-worth of html+js each time you open up the wiki can be quite slow.
As for lazy loading --- when enabled, it breaks the indexing (e.g. finding backlinks) and searching because these are done by the js inside the browser too. I only lazy-load images and binaries because they are not indexable anyway.
Either way, super powerful.
Refs: https://en.m.wikipedia.org/wiki/Datalog https://github.com/tonsky/datascript https://www.datomic.com/
The inline programmability of DataView and the Grid views etc are pretty killer. Unfortunately Obsidian is not for me, so I hope Logseq catches up.
I guess I fit solidly inside the 5GB free tier and probably will for quite a while.
I would use Dropbox but the app feels quite heavy and it wasn’t very easy to set up on iPhone.
* list all unfinished tasks grouped by some custom metadata ( project, etc.)
* have note templates which contains a dynamic query for the history of the same type of note for historical context
* use it in conjunction with Obsidian charts to make custom charts ( e.g. number of meetings with tag/metadata/mention of X)
Currently, I'm using Evernote and its new Tasks feature is very good, even though it could be better.
I cannot move from Evernote to Obsidian if I don't solve the issue of managing tasks effectively from with Obsidian, as I currently do with Emacs.
E.g. (note HN strips out emoji)
- [] Write documentation for Foobar #docs #personxyz (cal emoji) 2022-05-22
E.g.
```tasks
not done
description does not include #deferred
due before today
path does not include Templates
```
There are other plugins you can use, like Dataview which is the subject of this post (but means each Todo needs to be its own note), or the Todoist plugin, but Tasks is the one which works for me.
I understand that it is a complex thing though, and not something easy to just throw in
I wish it were FOSS, or had a roadmap to become FOSS once the whole world already owns a licensed copy, like everyone should, but even though I'm normally a major FOSS enthusiast, I'm just happy this software exists at all, in any form.
I will definitely be making use of this dataview extension!
Tag wrangler is the other.
Maybe there are some basic queries that could support "two-way binding" of this sort, but it would be fiendishly complex and ultimately better solved by a different tool altogether
It doesn't need to be between dataviews. Just temporary filter the view, or export it in a regular static table. This is just about enhancing the View, not changing the concept.
Which opens the question, can dataview also take tables as data-source?
> Maybe there are some basic queries that could support "two-way binding" of this sort
I'm not talking about Two-way-bindings. Though this would be awesome. And I don't think it would be that hard to implement in general, depening on how exactly obsidian works internally with dataview.