What Is a Query Planner?
planetscale.com
planetscale.com
So if you execute the query "SELECT * FROM (a, b) WHERE a.foo = b.bar", if you have many rows in `a` but few rows in `b`, then it's much for efficient to scan `b` than `a`. The query planner will keep track of properties like this & come up with tricks to speed up execution.
But in the sense that "everything is a compiler", yeah you could totally think of a query planner as a compiler that takes in your query's AST and a statistical description of your data and lowers it to a query plan.
The actual compilation of the plan to machine code is possible and a few database systems do exactly that. But most process then nodes themselves or JIT specific node types represent simpler or more tightly defined operations.
Compilers generally just make a lot of constant-factor improvements without changing the algorithm. One exception might be the tail-call optimization, which changes the space complexity of an algorithm. And that's one of the optimizations where the developer needs to know for sure whether it will br applied or not.
This seems incredibly intuitive. Anyone reading it would get the impression that query speed is very important for database users and that they would be eager to investigate any new technology that promised a substantial improvement in query performance.
Yet, I have found the opposite. I created a new kind of database that shows incredible performance gains when compared with mainstream databases like SQLite (https://www.youtube.com/watch?v=Va5ZqfwQXWI) for simple and complex queries against tables of all sizes. It is able to do this without needing to create separate indexes on all the columns in the WHERE clause. The software is in open beta and available for free download at (https://www.Didgets.com) but it is like pulling teeth to get people to try it out.
If I post something like this on HN, I usually get lots of comments about how my claims are impossible (or at least improbable) but few are willing to actually test it themselves on their own machine with their own data set. I have several videos that demonstrate how quick and easy it is to load and analyze data, but interest has been minimal so far.
I feel like the inventor of a new kind of car engine that claims to get 2x the MPG while delivering the same horsepower, but can't get anyone to take the free test drive and prove me right (or wrong).
It's natural to be skeptical of someone who is afraid to share their implementation details but claims best-in-class results, isn't it?
So...are you promising to try it out if I tell you how I did it?
I also never use tech that doesn't share "how it works".
Clear?
> "how it works" in quotes
Well, see, I can reason about SQL semantics via knowledge of B-trees, and I can understand how there's a time vs space tradeoff made in SQL implementations around indexing - add indexes to save time, but they write to disk.
Didgets... I don't even know what it is. Do I query it with SQL? Does it only work from your UI DB browser? I could watch videos to find out but I find you a bit combative, so my desire to bridge the gap from my end is pretty low atm
If someone tells you they’re faster than their well-funded, well-developed peers, then either:
1. They’re dramatically better engineers
2. They’ve constrained their data model to make it faster, likely at the cost of some usecases
3. They’ve developed some novel theory or set of tricks
4. Their benchmarks are limited/biased, or plainly wrong
5. They’ve gotten to drop backwards compatibility constraints their competitors couldn’t
With usually the answer being 2 or 5 if they’re legitimately faster.
In your case, based on glancing at videos
1. You seem to have widened the data model significantly
2. You’ve got no claims to novelty except the notion of didgets, which is an abstraction I’d expect to cost a pretty perf penny
3. You’ve got no claims to particular competency
So… I’m instinctively betting on 4.
But if my theory is right, you’ll have a lot more success getting people to test it if you resolve the other items. Which I think you could do in probably two paragraphs on the site.
Even if I could convince myself that it’s faster, I still wouldn’t be comfortable with it if I couldn’t explain, even at a high level, why it’s faster. And I would never assume it’s faster for anything except exactly what I tested (which inevitably is only somewhat representative of what I actually use it for)
Think of it this way - that kind of information is required for me to even consider something like this.
It is not yet a 'drop-in replacement' for an enterprise, distributed database. It can't yet do everything a file system can do. Instead, I have built a tool on the technology that can do some very interesting analytics against large DB tables. It can create pivot tables in seconds. It can help you find anomalies and clean your data quickly. But some other features like file indexing and log management features still need a lot of work.
If you have an immediate need for the polished features, then it can be very useful as it is. If you have a need for features that are only partially finished, then I need feedback, testing, and resources to get them working properly.
There are dozens, if not hundreds of products in this space. A popular example is Power BI, where even the desktop edition can produce extremely fast pivot table reports over about 10 GB of data.
The server edition can produce sub-second reports on terabytes of data, not a mere couple of gigabytes like in your demos.
You'd have to demonstrate a significant improvement over the existing state of the art for anyone to be interested.
For getting a quick overview of the advantages across it's probably easier to craft something compelling in writing.
I take data integrity very seriously and it sounds like you do as well. You would make a good beta site.
Whenever you offer an alternative to a mature technology like database, it can be a real challenge. Your product is expected to do everything that the well-funded and long developed product can do, and more.
Good luck on your venture.
I don't want to be critical but do want to give some advice. The fact that most information is embedded in videos detracts from the ability to understand your product. Kicking the tires on technology may be free but it's also a time investment thats competing against other priorities.
I have little sales and marketing experience so I haven't done enough yet to promote it with proper documentation and other literature. I thought that the videos were the best way to show its speed and flexibility. People can see with their own eyes how to do something and also see it isn't just vaporware. Clearly, by little startup needs a lot of help to compete with the 'big boys' even if the underlying technology is superior.
Duck DB's landing page has a lot of great lessons about how to pitch your idea so that it will land well. For one, notice that it is much more concrete than Didgets':
"DuckDB is an in-process SQL OLAP database management system" vs. "A WHOLE NEW WAY TO MANAGE ALL KINDS OF DATA: Beta testers wanted"
Any creator will like their idea because they can see the novel value. But other people probably aren't as interested that it's new. They want to know what it is in the most concrete terms possible.
One other thing: is anything stopping you from putting the results you showcase in your videos into a table on the homepage?
Elsewhere on HN you mention that you're looking for "innovators" and "early adopters", but your target market seems to be "database neophytes". Is there a market at the intersection of those two sets?
In other posts and comments across the internets, you don't seem to have decided if you're creating a database or a filesystem replacement. Once you have internal clarity on that, you'll be better at talking about it with people who might find value in what you're building.
The objects I invented to store the meta-data tags turned out to be very performant Key-Value stores that could also be used for other purposes. It turns out they could also be used as columns in a relational database table. I think I was surprised as anyone when queries against my tables that use them turned out to be substantially faster than for other databases with highly indexed tables.
I am sorry if the term 'prove me wrong' came across in a bad way. I guess I just got tired of people telling me my system couldn't possibly work and refuse to put forth any effort to try it out. In some ways I feel like a Tesla salesman surrounded by people who are sure it couldn't possible be better (or even different) than a regular car, but won't get in and take it for a spin around the block.
I don't want to poopoo your product, but the material you've provided does not give me confidence to try it. Claims are all very well, you need to show your working and get it right.
Something like postgres has a lot of people working on it, plenty who understand it and the various performance tradeoffs. Remember someone demoing how postgres could achieve much better performance than some competing models by turning off all the important stuff that makes it transactional and atomic and reliable.
> Claims are all very well, you need to show your working and get it right.
What does this mean? I have un-doctored videos that show various features in action. How do you suggest that I otherwise 'show your working and get it right'?
No, you haven't. You have found that so far, you have not convinced people to look into your product.
For reasons you are explaining yourself here: https://news.ycombinator.com/item?id=34008598
No matter how good your code is, it will go unnoticed without proper communication. Written. Especially if we cannot read the code.
> you don't have the time and skills necessary to import it into a big database system, then maybe we can help.
You are specifically telling people "This is for noobs, if you use a real DB you are not the target", so of course people with DB-related perf issues (and who most probably already have a "big database system") will pass. This is far from the only blocking issue on your website, in your HN comments and your HN description though.
But it does allow you to quickly import a data set into a table and do a variety of analytic or cleaning operations on it and it is easy enough to do that even a novice without any SQL skills could do it.
Our hope is that enough people will notice its strengths that we can get the resources needed to finish the remaining pieces.
As a fairly seasoned database application developer, I'm perplexed by your web site. It's very sparse and requires the reader to watch a bunch of videos. Even then, there's a lack of clarity about what the product does. Apparently this is some kind of Windows-specific desktop application that can load data and let you query it? Is that the only way? Is there a CLI version I can run on Mac or Linux? Is there a server version? What kind of data can it deal with? How many columns and rows does it scale to? And so on.
On the surface, a slick tool to quickly explore data "ad hoc" — as opposed to being the backend for an application running many types of queries — would be useful, but what exactly is the value proposition? It can't be the GUI, which seems rudimentary. You focus a lot on performance, but if I load a bunch of data into SQLite, it doesn't really matter to me if one query takes 100ms or 2000ms to run a query. That's a "one sip of coffee" level of annoyance that is not going to drive me to investigate alternatives. On the other hand, if I have gigabytes — or even terabytes or petabytes — then the problem becomes less tractable, but there are good solutions (PostgreSQL, BigQuery) that I can use to run complex queries in seconds, using my full set of SQL skills. Your web site gives me zero confidence that it's built for that kind of scale and that I would be productive using it in such a case. So what kind of sweet spot does it fill?
Honestly, I think you would get more traction by making it open source and spending more time just describing its possible utility, rather than complaining that people don't understand. The hacker community tends to judge by merit alone, not claims. It is your job to convince people; it's not people's job to convince themselves. As a developer, I'm mildly piqued by the prformance claims, and I would love to learn more about how it organizes and queries data, but I get really suspicious when all that's offered is a magical black box. Even commercial, closed-source databases like Oracle and SQL Server describe their internals in a fair amount of technical detail.
With all due respect, you also come across as combative and even somewhat rude here on Hacker News. If you're not getting the responses you want, maybe there's a reason for that.
The database engine is cross-platform and is capable of running in a number of environments, but so far we only build the Windows version of the browser application. The engine has an API that the browser app calls. Any other app could also call it, but so far we just have a few test programs that run outside the browser GUI. We built a Linux version and tested it, but at this point we just don't have the resources to build and support multiple platforms for every revision.
I have personally built tables with over 100 million rows and over 2000 columns, so it scales pretty well.
I definitely need to work on presentation materials so that developers can better understand what it is and where it can go. I really don't mean to come across as combative, it is just a bit frustrating when people criticize something that they have not personally tried.