Hacker News Reader

Hacker News

Comment by mgaunard | Hacker News Reader

Parent Full thread

mgaunard

1. Memory map a file

2. Manage access to it through a server

There you go, you know have a database.

View on HN

1. Draw some circles

2. Draw the rest of the owl

https://www.pillarsofimpact.com/post/draw-the-owl

Memory mapping is a poor way to make a database[0], not all storage can be memory mapped, and this doesn't provide any of the functionality you expect of a database like indexing and concurrency control.

[0] https://db.cs.cmu.edu/mmap-cidr2022/

I saw a headline about the book on Reddit and made a joke to a friend of mine: "How much you wanna bet it uses mmap?" Well...

It uses mmap.

The concurrency control is managed by the server that manages access.

All databases that are any different than these two points are just bad.

> Manage access to it through a server

This has real "draw the rest of the owl" energy.

How to make a DBMS? 1. Get a file 2. Make a DBMS

> 1. Memory map a file

Complex when you want to

> 2. Manage access to it through a server

A database is only "easy" IF:

- Append only

- No real "delete" or "updates" just to reiterate the above.

- Only Sequential scan

- Only need simple iterator-per-row

- No maintain secondary stuff like indexes, so not need to coordinate changes

- No concurrency

- Fit in RAM, and I mean in few MB

- No need to deal with SQL, use his own DSL (sql is so bad! so much weird stuff!, but is ok to have something sql-ish like LINQ)

- No need to deal with recursive data types, only scalars

- Is only embebed

- No need auth or security validations

Ok, after making this list, I sure forgot some other tips to make this easy!

There is no requirement to have the file fit in RAM to mmap it, that's the point.

Obviously you'd have a file per column (or index), and use directories to represent tables.

This is the correct way of doing it and yet so few databases do it.

"correct" is not an objective measure, it's a function of use case

mmap is not a panacea, it improves specific access patterns by incurring specific costs, it's definitely not true that mmap is the right choice for all databases

Yes, I understood, but "mmap a file" that should be a database is not "easy", because now you can deal with segfaults and other weird things that are out the normal playbook.

Of course, depending on the other things of the list this is or not a major issue. Is more about how combining several ideas leads to a easy or complex implementation.

I worked on a commercial product that did just this, except it also had a "transaction log" component. All access through the server was logged, and the log could be used for replay/recovery. While it worked for them for thousands of simultaneous clients, it was not a general purpose solution. There were no real indexes: the "indexes" (hash tables, really) were built in memory at startup.

i don't understand what you mean by no real indexes

why does generating indexes at start-up not count as having indexes?

(asking because i do this all the time)

I suppose I should've said "persistent" indexes. There was also only a single type supported: hash table. And it was hard coded to 2 fields. You had to change the code (which was C) to change the database.

This system was rarely restarted, so in practice it didn't matter what it did at startup, as long as it didn't take more than a few minutes, but it did place some limitations on data size. (This was a 32-bit system and everything was memory mapped.)

1. Build your own database

There you go!

You seem like a real go-getter who can cut through red tape and find simple solutions to complex problems :D

The power of abstraction!

why is this so heavily down-voted?

it feels correct, even though it's not a complete guide

(reliably persisting changes to disk is a big part of what dbs do, but is missing here)

Good to now.

Reply on news.ycombinator.com