LittleD – SQL database for IoT, can run queries in 1KB

Is it comparable with binary trees [O(n log n)] and hashtables [O(1) on average]?

    /* Advance a pointer num_times times.  Notice, this does no checking, so
    you better know what YOU ARE DOING!!!! */

As you noted, its not really an IoT databases. It could be used as such, but thats not the reason I built it.

I have also been working on the data transfer problem, because without the ability the share the data it is in fact kind of a useless platform in many applications. I have a job manager being developed that will allow for scheduled or ad-hoc execution of functions. I've also written some small library to encode LittleD results for network transmission. Using some basic networking stack, it would be easy to assemble these pieces into something that could be viewed/controlled from a centralized platform. LittleD could even be modified with relative ease to actually query over other LittleD instances!

EDIT: I would actually like to encourage you to divulge some specific criticism for the IoT application. Are you specifically speaking to your automation system, or to IoT at large? It seems difficult to predict exactly how any one person might apply any given technology.

Frequently it is exactly when connectivity is gone that you want some intelligence combined with data upon which to act, or a buffer in which to spool data. Otherwise you've just got a pile of sensors that are slightly easier to deploy than the ones with long extension cords.

Put another way,

> If your data is at all valuable/useful, you don't want it sitting out somewhere on some device that will hopefully/maybe be online when you go to query it.

So what do you do with data on your device when it is offline? If it is at all valuable/useful, hopefully you're not just dropping it on the floor when Daddy-node is unreachable. Sounds like a job for which some sort of structured data store might be useful...

What if these IoT devices aren't just dumb sensors, and need to control some process? Then your argument is reversed; you want the data to be available locally, instead of having to contact a remote server that "will hopefully/maybe be online".

Any time you want to store historical data, and query that data later.

Very similar to how you might imagine using a database for a website, just at a smaller scale. The project that inspired this work was a water metering project in Kelowna, BC. Basically, a friend of mine took a bunch of micro-controllers with soil moisture sensors, shoved them in the ground, and tried to come up with better demand-based watering schedules. It took them two months to develop the data management code, and knew it could have been days or hours with a proper database.

As for the name, I cannot take credit. There is a good story behind it though. ;)

As for the need, I sort of explained the motivation this comment: https://news.ycombinator.com/item?id=10622675. Data management would drastically reduce the development effort associated with data-intensive applications for smaller devices, in an IoT setting or otherwise. I've actually recently travelled to the University of Michigan to talk about this work, and got lots of good feedback. One of the professors invited me and has asked me to study with him, because he feels there is a need. One of his graduate students there is working on integrating LittleD into some of his work already!

For example, https://www.sqlite.org/about.html claims that SQLite can run in "very little heap (100KiB)"; this project targets an Atmel ATmega2560 which has only 8KiB of RAM.

(I doubt "5 times smaller than SQLite" is a useful comparison; the limiting factor is probably RAM consumption, as opposed to executable code which can be stored in cheaper non-volatile memory.)

The corresponding paper [1] claims "SQLite requires at least 200 KB of code space and cannot run on popular platforms such as the Arduino." For comparison, Arduinos based on ATmega microcontrollers only offer up to 256 KB of flash memory at maximum.

[1] https://dl.acm.org/citation.cfm?id=2554891

But that's not the biggest problem. SQLite, like most programs, relies on having a C library like glibc available. This is pretty much the intermediary person to the operating system - you can ask it for memory, it will allocate memory for you, you can ask it to open a file, it will facilitate that. Most programs assume it's always present and so the dependency isn't properly abstracted. This is a problem when you don't have an operating system.

Notice also that you can't just go and implement or stub everything that glibc does. To do that, you would just end up having to add an operating system.

(This particular rabbit hole goes much deeper. A lot of programs sadly don't depend on a C library, they depend on glibc in particular. This causes massive problems for systems that run Linux yet can't accomodate glibc, like OpenWRT. They use an alternative C library that works fine 99% of the time, and for the remaining 1% it will cause programs to silently misbehave or crash because of an implicit dependency on whatever glibc does. I had this problem with the hwclock utility which started segfaulting when OpenWRT switched to musl libc)

SQLite itself requires at least 100KiB of Heap space, sometimes many times more memory than already available: http://www.sqlite.org/about.html

Granted. That was 10 years ago, but it was also quite a beefy device for its time. So while naturally the devices have become quite more powerful in the last 10 years, current IoT devices are also much more compact than the old ck1.

As such I would say it's conceivable that you don't have the memory to easily run current SQLite on any of these devices.

SQLite is probably on of the most well tested piece of software on this globe. I do not know if this space saving is worth it.