TIL–Python has a built-in persistent key-value store (2018)
remusao.github.io
remusao.github.io
[0] https://docs.python.org/3.10/library/shelve.html
Obligatory pickle note: one should be aware of pickle security implications and should not open a "Shelf" provided by untrusted sources, or rather should treat opening a shelf (or any pickle deserialization operation for that matter) as running an arbitrary Python script (which cannot be read).
https://github.com/cristoper/shelfcache
It works despite some cross-platform issues (flock and macos's version of gdbm interacting to create a deadlock), but if I were to do again I would just use sqlite (which Python's standard library has an interface for).
Yeah, I tried to use shelve for some very simple stuff because it seemed like a great fit, but ultimately found that I had a much better time with tortoise-orm on top of sqlite.
If you need any kind of real feature, just use sqlite.
And you don't even have to forget about Python 2! If you use format version 2 you can pickle objects from every version from Python 2.3+ and all pickle format are promised to be backwards compatible. If you only care about Python 3 then you can use version 3 and it will work for all Python 3.0+.
https://docs.python.org/3/library/pickle.html#data-stream-fo...
The reason against using pickle hasn't changed though, if you wouldn't exec() it, don't unpickle it. If you're going to send it over the network use MAC use MAC use MAC. Seriously, it's built in -- the hmac module.
It's best to use another dedicated type or library specifically for this task.
CREATE TABLE store(key TEXT PRIMARY KEY, value TEXT)
sqlite
Took 0.032 seconds, 3.19541 microseconds / record
dict_open
Took 0.002 seconds, 0.20261 microseconds / record
dbm_open
Took 0.043 seconds, 4.26550 microseconds / record
sqlite3_mem_open
Took 2.240 seconds, 224.02620 microseconds / record
sqlite3_file_open
Took 7.119 seconds, 711.87410 microseconds / record
sqlite
Took 0.040 seconds, 3.97618 microseconds / record
dict_open
Took 0.002 seconds, 0.19641 microseconds / record
dbm_open
Took 0.042 seconds, 4.18961 microseconds / record
sqlite3_mem_open
Took 0.116 seconds, 11.58359 microseconds / record
sqlite3_file_open
Took 5.571 seconds, 557.13968 microseconds / record
Fixing that on my machine took the sqlite3_file_open benchmark from 16.910 seconds to 1.033 seconds. Adding the index brought it down to 0.040 seconds.
Also, I've never really dug into what's going on, but the dbm implementation is pretty slow on Windows, at least when I've tried to use it.
Sure, mutating data sets might be a useful use case. But, inserting thousands of items at once one at a time in a tight loop, then asking for all of them is testing an unusual use case in my opinion.
My point was that we're comparing apples and oranges. By default, I think, Python's dbm implementation doesn't do any sort of transaction or even sync after every insert, where as SQLite does have a pretty hefty atomic guarantee after each INSERT, so they're quite different actions.
Seems like this would be why: https://news.ycombinator.com/item?id=32852333
No kidding.
https://github.com/python/cpython/blob/main/Lib/dbm/dumb.py
It uses a text file to store keys and pointers into a binary file for values. It works .. most of the time .. but yeah, that's not going to win any speed awards.
In the 90s you might target dbm for your portable Unix application because you could reasonably expect it to be implemented on your platform. It lost a lot of share to gdbm and sleepycat's BerkeleyDB, both of which I consider its successor.
Of course all of this is now sqlite3.
> dbm is a generic interface to variants of the DBM database — dbm.gnu or dbm.ndbm. If none of these modules is installed, the slow-but-simple implementation in module dbm.dumb will be used. There is a third party interface to the Oracle Berkeley DB
I don't know how fast or slow the "dumb" implementation is, but I can bet it is way slower than gdbm. I can see someone using this module, considering it "fast enough" on GNU/Linux and then finding out that it's painfully slow on Windows and shipping binaries of another implementation is a massive PITA.
- it backed up by sqlite, so the data is much more secure
- you can access it from several processes
- it's more portable
I was just pointing out that the title and even the article seem to associate this system with python per se, rather than understanding it as python's interface to a common and pre-existing system.
I think this is because each time you call execute (which is probably sqlite3_exec under the hood), your statement is prepared again, and then deleted, while with executemany it's prepared once and then used with the data. According to the SQLite3 documentation of sqlite3_exec:
> The sqlite3_exec() interface is a convenience wrapper around sqlite3_prepare_v2(), sqlite3_step(), and sqlite3_finalize(), that allows an application to run multiple statements of SQL without having to use a lot of C code.
I knew that when you execute a request a lot, prepared statements are faster, but it seems that it's not exactly the case and that all statements are prepared, the performance improvements come from preparing (and deleting) only once. The documentation page about the Prepared Statement Object has a good explanation of the lifecycle of a prepared statement, which also seems to be the lifecycle of all statements.
As others have pointed out, it's the lack of a suitable index that's the real problem here.
property runCount : 0
set runCount to runCount + 1
https://developer.apple.com/library/archive/documentation/Ap...
PRAGMA journal_mode=WAL;But that only works if all given processes run on the same physical machine, and not in containers/jails/VM.
The only namespace that matters here is the filesystem/mount namespace.
There is no reason you can’t access the same shared SQLite database on a common volume between containers on Linux for instance.
That is false. Whether you mmap the same file is dependent on the specific SELinux policy - which is by design, highly configurable.
I am also surprised/skeptical that the default configuration for those OSes for a shared volume/mount point would disallow mmaping by default (which is all that is required) - since what is the attack vector they’re preventing? If you can exfiltrate via memory you could just do so via read/write.
You'll either have to reduce safety and usability, or drop SQLite.
There are situations where you wouldn't want to, but they are probably very uncommon outside of the embedded computing world. Copying 3 files vs 1 is not a gigantic deal. Most of the time you aren't even moving a SQLite database around.
Of course, you'd be better off with WAL-based replication tech like litestream instead of plain file copies if you are truly worried about parallel operations during your file copies.
I used sqlite in the projects name. This makes me think of an article I read recently that suggested not using descriptive names for projects. For reasons such as this.
cat foo.csv | sort | uniqif you go and look at the binary database you'll see random other bits of memory in there, because it doesn't initialise most of its buffers
My money is that’s mostly because it actually stores the data. I don’t think dbm guarantees any of the data written makes it to disk until you sync or close the database.
This writes, guesstimating, on the order of 100 kilobytes, so chances are the dbm data never hits the disk.
That said, what database dbm is using varies a lot by platform. I think on linux it's usually BerkleyDB.
I use dbm a lot to cache pickles of object that were slow to create. eg pandas dataframes
But... It's Oracle...
(simplification) With any time you are mapping some file to some data structure (dbm, sqlite, mmap), the file descriptor generally stays open while you are working with it, meaning reads/writes/seeks are just about as fast as IO can go.