How to use supervised learned classifiers without a training set by using LLMs.
127 karma · joined October 7, 2019
For the index structure itself you can use succinct dynamic data structures to reduce the size. The bulk loading approach is particularly amenable to use of succinct data structures as the node-vectors in a HNSW are montonic they can use an Elias-Fano encoding. The neighborhoods can use log-arrays.
Creating a multi-index also seems possible, where you use the triangle-inequality to prune the candidate vectors. This will probably require storing the distances for neighbor distances in the bottom layer, in order to be time-efficient for query and would thereby be slightly bigger. I haven't tried it yet but I intend to.
There is also the possibility of using random projection for dimensionality reduction. I also haven't tried this yet either but will give it a go soon. We haven't folded the parallel hnsw into our open source VectorLink yet, but we'll be doing it in the near future - we wanted a little bit of stability of approach first.
Diffs can be constructed between two objects, or you can get sets of diffs of objects between commits automatically. You can manually construct diffs and use them to patch branches.
We don't have a conversion tool from SQL database tables, but it's something on my list.
You should give TerminusDB a go (https://terminusdb.com/), it's really OSS, the cloud version is cheap, fast, there are not tons of baroque settings, and it's easy to backup using clone.
TermiusDB is a graph database with a git-like model with push/pull/clone semantics as well as a datalog.
CMS stands for "content management system" and headless means API-based, with no restrictions over where you use the content. Devs are you folks.
Existing headless CMS tools sometimes make it up as they go along - starting with the idea of ‘I want to build a company that delivers a headless CMS’ and then quickly slapping a bunch of technologies together and ending up with a pile of JSONs floating around a MongoDB or another similar frankenstein. As we’d already built the document graph data layer from the ground up, we could properly integrate the CMS features to give a seamless developer experience. We are never going to send you to a screen that says, ‘TerminusCRM requires NodeJS version 10+ and a Mongo database’. It is all contained in one in-memory, highly compressed data management system that we designed and built for this specific purpose.
A few of the other offerings are just headless markdown backed by git - which is actually a good idea, but comes with the capacity and performance limitations that git implies.
Why not have a highly performant document graph content management system that incorporates the most important concepts from git in the data layer? All the version control features that content needs (clone, push, pull, branch, revert, merge) and are highly awkward in other systems.
We also thought that GraphQL was the obvious data manipulation choice for content, but found weak implementations wherever we looked. For TerminusCMS, we’ve implemented a suite of features which allows you to query a TerminusCMS project using GraphQL in such a way that deep linking can be discovered. We can use path queries with GraphQL.
To summarize our market thoughts - it seems that devs want: * Deploy anywhere open-source * Dev-first in memory, highly compressed, scalable and fast content management so you can build complex and fully featured web apps of every shape and size * Git-like features to help with permissions and version control (and - crucially - merge) * Best-in-class GraphQL implementation
And there was nothing offering that mix until we released TerminusCMS.
TerminusCMS is a content platform that sits at the convergence of content and knowledge. It is a model-driven, API-first approach to content management. With TerminusCMS you can use your data as content. The data employed in content has meanings associated with it, and because that content is well structured, the content can also be used as if it were data — in fact, that content is data. TerminusCMS is structured like Git, so you get all the git-like features for your content engine. History, change management, branching, non-linear development, easy backups, distributed development, and more. You can enrich your content with semantics which can give you the ability to personalize content and build superior recommender and AI/ML systems.
Content curation needs change requests: you need to be able to add a new translation, or new content in a branch, which is viewable as an entire site, but which only goes into production when you "merge to main". TerminusCMS gives devs a powerful way to define schema, query, deliver content and assets to front end. It has schema-as-code; is standards-based for interoperability; has an extremely fast publishing API; and provides workflows made by way of content and network model, not wired into product itself. It give editors a way to manage content automatically so that devs don't need to do much to support them. You can use TerminusCMS to build your complex app, or to manage your organization knowledge.
TerminusCMS is open-source all the way down, so if we vanish or piss you off, you can recover and continue.
The business is cloud hosting and enterprise delivery, which we hope can keep the wolf from the door. It is freemium with a generous free tier and easily cloned examples, so take a look.
TerminusDB uses a datalog under the hood, but GraphQL can expose a lot of datalog like power with the right interface. Ultimately we want to have named parameteric queries in datalog exposing "pseudo-edges" in GraphQL which can be treated just like a normal field but are actually calculated.
And in the future we will even be able to add constraints - which can be a real superpower in querying graphs.
However, it's not just a question of whether one can be made into another. The use of contexts is very cumbersome, since you need to specify different contexts at different properties for different types. It makes far more sense to simply have a schema and perform the elaboration from there. Plus without an infrastructure for keys, Ids become extremely cumbersome. So beyond just type decorations on the leaves, It's the difference between:
{
"general_variables": {
"alternative_name": ["Sadozai Kingdom", "Last Afghan Empire" ],
"language":"latin"
},
"name":"AfDurrn",
"social_complexity_variables": {
"hierarchical_complexity": {"admin_levels":"five"},
"information": {"articles":"present"}
},
"warfare_variables": {
"military_technologies": {
"atlatl":"present",
"battle_axes":"present",
"breastplates":"present"
}
}
}
{
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11",
"@type":"Polity",
"general_variables": {
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/general_variables/GeneralVariables/e4360ee3766c2863f06a34ffcdd9869d41b03d04c6f6af5f94b0a14a47e8e704",
"@type":"GeneralVariables",
"alternative_name": ["Last Afghan Empire", "Sadozai Kingdom" ],
"language":"latin"
},
"name":"AfDurrn",
"social_complexity_variables": {
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/social_complexity_variables/SocialComplexityVariables/191353c4b7138842ec4029dd07fbd63c9dda752f0cd72b1584f046a274cf024c",
"@type":"SocialComplexityVariables",
"hierarchical_complexity": {
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/social_complexity_variables/Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/social_complexity_variables/SocialComplexityVariables/191353c4b7138842ec4029dd07fbd63c9dda752f0cd72b1584f046a274cf024c/hierarchical_complexity/HierarchicalComplexity/d6a772c5c6919cc511a24ab89f908032aa32b1e3e939d2e0c32044b3a5d9151d",
"@type":"HierarchicalComplexity",
"admin_levels":"five"
},
"information": {
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/social_complexity_variables/Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/social_complexity_variables/SocialComplexityVariables/191353c4b7138842ec4029dd07fbd63c9dda752f0cd72b1584f046a274cf024c/information/Information/2f557c1016552f30b8d8bb1bdd9a8584791dd06d32f25bded86a7eb59788ea7f",
"@type":"Information",
"articles":"present"
}
},
"warfare_variables": {
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/warfare_variables/WarfareVariables/704a2c1854a2fe80616fbea0ef0dcd6ce47f5174529ca191617e42397108c437",
"@type":"WarfareVariables",
"military_technologies": {
"@id":"Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/warfare_variables/Polity/7286b191f5f62a05290b8961fd8836a26ddc8399611b216fae4aaacc58ba6c11/warfare_variables/WarfareVariables/704a2c1854a2fe80616fbea0ef0dcd6ce47f5174529ca191617e42397108c437/military_technologies/MilitaryTechnologies/80a91b3e5381154387bde4afc66fdd38834de16c671c49c769f5244475cbbb1b",
"@type":"MilitaryTechnologies",
"atlatl":"present",
"battle_axes":"present",
"breastplates":"present"
}
}
}I'm going to have to go away and think about how one could effectively leverage this in a data setting, but I'd love to hear ideas.
Echoing what triska said, CLP(ℤ) and friends are some of the most under-appreciated aspects of prolog implementations.
I'm amazed that programmers still don't have access to CLP when trying to do scheduling and planning solutions.
As an example in practice, what if you want to know about a transaction in which a number of entities transitively had holdings in one of the beneficiaries of the transaction at that particular time. The date window is not known, and the date windows are important in the ownership chain as well as the transactions that are being undertaken.
With CLP(FD) you can ask for a window of time, and the solution will zoom in on an appropriate time window which exists for the entire chain and match the time of the transaction.
Now try to do this query in SQL. It's almost impossibly hard.
I can't wait until I have the time to implement constraint variables for TerminusDB, but at the minute we are still working on more prosaic features.
Aside from that there are very interesting program correctness and optimisation systems which are based on prolog (usually a datalog). For instance Soufflé: https://souffle-lang.github.io
> If anything the lesson from git's success should be that the aurhor's approach (the diff/patch way of Subversion etc.) is exactly the wrong one.
The problem with subversion was not the mechanism of storage of objects. And the strength of git doesn't lie here either. It's rather the very flexible nature of its commit metadata and it's design which builds on the idea of multiple masters. This was what was stifling in subversion.
The method of storage whether of the deltas, or the individual states is trivially equivalent. You can inter-convert them if you'd like, so clearly this can't be a game changer.
But even if it were and git's approach was "The Right Thing", this would not be a lesson to learn from git when dealing with data storage because it scales poorly.
Every state transition would have to re-represent the entire state of the database. This will work fine for code, but for databases with tons of objects changing and being extended all the time it would be crazy. It makes a lot more sense to store the differences.
I write a little bit more about the relationship here: https://github.com/terminusdb/technical-blogs/blob/main/blog...
Moving to LGPL doesn't improve anything with regards to the non-problem of using the GPL for this purpose in the first place.
The problem lies outside of the technical/theoretical legal entitlements.
We want people to build stuff on our database whether they are companies or individuals and we don't want to encumber this with stumbling blocks.
We intend to do this by having a SaaS revenue model which doesn't require we hobble anything in the database in order to make money.